Friday, September 18, 2009

Facepalm supreme - Deceived by a button

Okay, so I've produced a piece of code looking like this. It's the peak, the epitome of simplicity, no doubt about it.

// We only want to be able to delete the item if we either:
// [1] have accesslevel 0 (admin privs), or
// [2] are the author of the item (identified by session and database id)

if(isset($_GET['delete_id']){
   $delete_id = // Sanitation removed for simplicity

   $author_id = // DB operations removed for simplicity

   if($_SESSION['accesslevel'] == 0 || $author_id == $_SESSION['user_id']){

      if(mysql_query("DELETE FROM table WHERE id=$delete_id") or $output = 'Error removing item '. $delete_id)
         $output = 'Item ' . $delete_id . 'deleted!';
   } else {
      $output = 'You do not have the privileges required to remove this item!'
   }
}


Then we have a couple of unrelated clauses, and the output of the list of items, which of course displays a possibility to delete the item if the privileges are upheld. It looks like follows:

$result = // Query for all relevant items

while($row = mysql_fetch_array($result)){
$button_delete = ''; // Empty if unprivileged
   if( /* $_SESSION['user_id'] is privileged enough */){
      $button_delete = "<a href='?page=$_GET[page]&delete_id=$row[id]'><input type='button' value='Delete'></a>";

   }
   echo "<tr><td>$row[item_name]</td> [...] <td>$button_delete</td></tr>";



// And finally the code to output the deletion-status, down here
echo $output;

Assuming all the code I decided to leave out are all without any errors and definitely doesn't interfere with the code above, can you find any errors?
My original source-file contained roughly 150 lines of code, and I was baffled when I found a very, very strange bug.

  • If I'm not privileged enough, I get the correct message that I lack privileges and the query is, of course, not executed.
  • If I'm privileged, the query is indeed executed, meaning we've entered the first IF-clause. ALAS, I get the same message as above (!!)


We simultaneously exeute the content of both the IF and the ELSE clauses. Whoa.


Isolating only the logical comparison part and running it, I was unable to reproduce the error. So some other part of the code had to be interfering. But there were no conflicts, there *could not be* any conflicts. Commenting my way through the code, I was assured of that. After spamming my code with outputs in the different clausules, (which only led me closer to the brink of insanity, since even though the code seemed to evaluate true (the query was executed), I did not get any output), the error was at long last stumbled upon.


<a href='?page=$_GET[page]&delete_id=$row[id]'><input type='button' value='Delete'></a> This here, ladies and gentlemen, was the perpetrator. The caused me such grief. Even though it was not supposed to have any effect on the code's workings (it was not in a form, it's value was never registered), it somehow managed to set some property amiss. Now, I'm not very well versed in the Document Object Model, but that my querystring was somehow "invisibly altered", and on top of that, that my evaluations were compared and their respective clauses were exectued before some variables were set (or set over again, as it appeared...), sounds very unreasonable, or at the very least, odd.



Just a short note regarding the debugging. For validating a user as author, I performed a simple query selecting the author's ID, storing it in a variable. When outputting this variable, before any DELETE-queries or basically anything at all was exectued, with unfulfilled privileges, I got the output correctly. But, when outputting it when it was true, it somehow was empty. Like the post had been deleted before the query was executed.
I'll write that again.


  1. Get author ID
  2. Echo this ID for debugging
  3. Compare to current user
  4. If true, perform DELETE
  5. If false, set errormessage


If #3 was false, we got the errormessage, and the author ID was echoed.
If #3 was true, we STILL got the errormessage, but the author ID was empty. The post was deleted before we performed the evaluation allowing the program to delete it, but ONLY when we actually DID have the permissions to do so.
My head still hurts...


Got any explanations to this? I'd love to hear about it!

MySQL triggers - automated magic

Working on a tenzui software project, I found a need to register every occuring change in the database, and it was not a hard task to realize that adding code for such monitoring to every single page, including ones generated by the client later on, would be cumbersome indeed. After some code-site traversing, I had my solution worked out. The glory of the open-source MySQL database shines through again - with a simple statement, a trigger, any desired action could easily be caught and acted upon. Diving in:

DELIMITER //
CREATE TRIGGER on_delete_from_archive
AFTER DELETE ON archive
FOR EACH ROW BEGIN
INSERT INTO changelog (action, table_name, item_name, user_id)
VALUES ('DELETE', 'archive', OLD.name, $_SESSION['userID']; END//
DELIMITER ;

A very simple, yet powerful example.

Suppose we have a table named archive, holding information about meeting minutes, publicly available documents etcetera. In your organization, you have a limited number of people who are able to add and remove content to this table. All good, until that day someone sets her UI password to "god," and pandoras box is opened. Now, some pesky kid playing deity quickly vaporizes your collection of .pdfs, leaving you devastated. Or not really, because you've got your backups, of course. But how did it happen?

By checking your changelog table, you can now see who and at what time any post was deleted from your archive table. Pinpointing the error to the haphazardly protected user, it's a small task to fix the hole. All with just a few lines of SQL, acting over all scripts making changes to the specified table. I find it pretty useful.

So, what does TRIGGERS support, really?
A trigger acts on the execution of any INSERT, DELETE or UPDATE statement upon the specified table. The event you want performed can take place either BEFORE the query has been run, or AFTER it. It also has the ability to handle "passing data," that is, the OLD and NEW data (which also happens to be the keywords for said operations).

These statements gives you ample opportunity to perform comparisons on two related pieces of data before performing the query activating the trigger, or as in our example above, log the successful execution of the query, together with related interesting data.

A thing of importance to take notice of is that triggers *only* react to SQL statements. APIs etc. can perform tasks on the tables without "triggering."

As you saw above, a trigger is created by invoking the "CREATE TRIGGER" statement. Up until recently, one would require the SUPER privilege to execute this, but with MySQL version 5.1.6, a new GRANT was added for this purpose: GRANT TRIGGER.

GRANT TRIGGER ON table.database TO 'user'@'localhost'

Thanks for your time! I hope you found these short notes useful, and that you'll make use of the awesome TRIGGER-statement in the future.

For more reading up, I'd recommend the following links:

Sunday, September 13, 2009

Guide to Postfix+Dovecot+MySQL Mail Server

Last week I spent a lot of hours researching and experimenting with Postfix (used to deliver mail, essentially an SMTP server) and Dovecot (POP and IMAP server, for handling the mails) trying to set them up properly with virtual users in a MySQL database. In this guide I'm going to go through the steps I ended up with in an easy-to-follow manner without going into too much detail. The systems I used are Slackware 12.1 and Slackware64 13.0 with Postfix 2.6.5 available at postfix.org, Dovecot 1.2.4 available at dovecot.org and MySQL v>5 available at mysql.com. I have yet to set up SSL and things like that, but once I do I might edit this guide or post a new one for that purpose.

Installing

First of all we need to install all the software we need. Note that I assume you will use root priviliges at the right times, such as when installing or starting/stopping a service. I won't explicitly say when you have to be root.
MySQL
I'm going to assume you have MySQL and it's libraries installed already, if not you'll have to find a separate guide for that (there should be plenty around).
Postfix
Download the tarball of the latest version (currently 2.6.5) from postfix.org and unpack with tar xvzf postfix-2.6.5.tar.gz. Go into the folder with cd postfix-2.6.5. Something that slightly annoys me here is that Postfix doesn't use flags to a configure script to compile with e.g. MySQL, but instead you have use a make command with unintelligible arguments before you do the compiling. To compile Postfix with TLS, MySQL support and Dovecot SASL (I think ;)) use this command:
make -f Makefile.init makefiles 'CCARGS=-DUSE_TLS -DHAS_MYSQL -I/usr/include/mysql -DUSE_SASL_AUTH -DDEF_SERVER_SASL_TYPE=\"dovecot\"' 'AUXLIBS=-L/usr/lib/mysql -lmysqlclient -lz -lm -lssl -lcrypto'
You might have to change the paths to what your system is using, those are what I used on Slackware and they seem to work just fine.

Before we can compile we have to make the user and groups Postfix will use. I made them with the following IDs and names:
groupadd postfix -g 2000
groupadd postdrop -g 2001
useradd postfix -u 2000 -g 2000

Now that we've done all that we can start compiling. Run make and make install. When it gets to the end it'll ask a lot of questions - you can just use the defaults (make sure the user and group is set correctly) unless you have any paths you want to change. After the make install is done Postfix should be installed and you can start it with postfix start to see if it works. Right about now it's probably a good idea to have a terminal with tail -f /var/log/maillog open. In there you will see all errors you get, in case you get any.
Dovecot
Download the tarball of the latest version (currently 1.2.4) from dovecot.org and unpack with tar xvzf dovecot-1.2.4.tar.gz. Go into the folder with cd dovecot-1.2.4. Fortunately Dovecot is kind enough to provide a configure script, which makes it easy to install with MySQL support. Run ./configure --with-mysql --with-sql-drivers. Now all you have to do is run make and make install and you should have Dovecot installed. You may want to create a dovecot user right about now:
groupadd dovecot -g 3000
useradd dovecot -u 3000 -g 3000
Try starting Dovecot with the command dovecot and check your terminal with maillog for any errors.

Configuring

Now that we've got all the software installed all that is left is to configure them! There's a lot more to do here than in the installing part, but it's nothing difficult so just follow along and it shouldn't be a problem.
MySQL
It's in MySQL we are going to store all the e-mail addresses and passwords and stuff we're going to have on our server. To get basic functionality what we need is a table for domains, a table for aliases and a table for mailboxes. We also need a user to connect with and a database to store the tables in. To create them log into mysql as root (mysql -u root -p) and issue the command CREATE DATABASE mail. To create a user called mail, with the password mail, that has full privileges on the database mail, run this command:
GRANT ALL ON mail.* TO 'mail'@'localhost' IDENTIFIED BY 'mail';

Without going into too many details I'll post the specifications of the tables below. You can just paste them into a file and use mysql -umail -pmail mail < mail.sql to have it create the tables for you.
domain
CREATE TABLE `domain` (
`id` int(11) NOT NULL auto_increment,
`domain` varchar(255) NOT NULL,
`transport` varchar(255) NOT NULL default 'virtual',
`active` tinyint(1) NOT NULL default '1',
PRIMARY KEY (`id`),
UNIQUE KEY `domain` (`domain`)
) ENGINE=InnoDB;
alias
CREATE TABLE `alias` (
`id` int(11) NOT NULL auto_increment,
`alias` varchar(255) NOT NULL,
`target` varchar(255) NOT NULL,
`domain` varchar(255) NOT NULL,
`active` tinyint(1) NOT NULL default '1',
PRIMARY KEY (`id`),
KEY `domain` (`domain`),
CONSTRAINT `alias_ibfk_1` FOREIGN KEY (`domain`) REFERENCES `domain` (`domain`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB;
mailbox
CREATE TABLE `mailbox` (
`id` int(11) NOT NULL auto_increment,
`username` varchar(255) NOT NULL,
`password` char(32) NOT NULL,
`maildir` varchar(255) NOT NULL,
`domain` varchar(255) NOT NULL,
`active` tinyint(1) NOT NULL default '1',
PRIMARY KEY (`id`),
UNIQUE KEY `username` (`username`),
KEY `domain` (`domain`),
CONSTRAINT `mailbox_ibfk_1` FOREIGN KEY (`domain`) REFERENCES `domain` (`domain`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB;
Postfix
The default Postfix configuration file has got a ton (I really mean A LOT) of comments that we don't really need. The final configuration file I ended up with is only about 15 rows. I removed pretty much everything and wrote one not far from scratch, you might want to do the same. If you prefer going through those hundreds of lines of comments, be my guest, though. :) The file in question is /etc/postfix/main.cf. Use your favorite editor (such as vim or emacs) and open it up. I'll just post my main.cf and leave any additional configuring you might want (such as different delays or a different greeting etc.) to you. The lines you have to change are the ones with myhostname = server.domain.tld and mydomain = domain.tld. Additionally you might need to add an SMTP relay since almost all ISPs block port 25 (and instead provide an SMTP server for you to use). I've commented out the line, remove the hash sign and add the URL to your ISPs SMTP relay server if you need it. Note that I removed a lot of lines where the setting was the default, see /etc/postfix/main.cf.defaults.
main.cf
myhostname = server.domain.tld
mydomain = domain.tld
myorigin = $mydomain
mydestination =
alias_maps = hash:/etc/postfix/aliases
alias_database = hash:/etc/postfix/aliases
#relayhost = smtp.isp.com
smtpd_recipient_restrictions = permit_mynetworks, reject_non_fqdn_hostname, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unauth_destination, reject_unauth_pipelining, reject_invalid_hostname
disable_vrfy_command = yes
virtual_mailbox_base = /var/mail/virtual
virtual_minimum_uid = 2000
virtual_uid_maps = static:2000
virtual_gid_maps = static:2000
virtual_alias_maps = mysql:/etc/postfix/mysql_alias.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql_mailbox.cf
virtual_mailbox_domains = mysql:/etc/postfix/mysql_domains.cf
Next we need to create the MySQL configuration files for Postfix (the three listed above). They tell Postfix where to look for aliases and users and their directories.
mysql_domains.cf
user = mail
password = mail
dbname = mail
hosts = localhost
table = domain
select_field = domain
where_field = domain
additional_conditions = and active = 1
mysql_alias.cf
user = mail
password = mail
dbname = mail
hosts = localhost
table = alias
select_field = target
where_field = alias
mysql_mailbox.cf
user = mail
password = mail
dbname = mail
hosts = localhost
table = mailbox
select_field = maildir
where_field = username
additional_conditions = and active = 1
That's all there is to configuring Postfix. :)
Try restarting to see if the new configuration works.
Dovecot
The Dovecot configuration file is similar to Postfix in the way that it's got a whole ton of comments. I got rid of them and I'll simply post my configuration file here. It should work without any changes. Note that mine was located in /usr/local/etc/dovecot.conf, not /etc/dovecot.
dovecot.conf
base_dir = /var/run/dovecot/
listen = *
syslog_facility = mail
ssl = no
disable_plaintext_auth = no
login_dir = /var/run/dovecot/login
login_chroot = yes
login_user = dovecot
first_valid_uid = 2000
first_valid_gid = 2000
protocols = imaps imap pop3s pop3
mail_location = maildir:/var/mail/virtual/%d/%n
auth default {
mechanisms = plain
user = root
userdb sql {
args = /etc/dovecot/dovecot-mysql.conf
}
passdb sql {
args = /etc/dovecot/dovecot-mysql.conf
}
socket listen {
master {
path = /var/run/dovecot/auth-master
mode = 0600
user = dovecot
}
client {
path = /var/run/dovecot/auth-client
mode = 0660
user = dovecot
group = dovecot
}
}
}
The longest configuration file so far. I haven't optimised it by utilising default values or anything, could be why. Next we have to set Dovecot up to know how to use the MySQL tables. It's done with /etc/dovecot/dovecot-mysql.conf as such:
dovecot-mysql.conf
driver = mysql
connect = host=localhost dbname=mail user=mail password=mail
default_pass_scheme = PLAIN-MD5
password_query = SELECT password FROM mailbox WHERE username = '%u'
user_query = SELECT maildir, 2000 AS uid, 2000 AS gid FROM mailbox WHERE username = '%u' AND active = 1
That should be all the configuring you have to do, so try restarting Dovecot with pkill -HUP dovecot and see if it'll start and connect to MySQL properly!

Setting up users

If everything is working all that's left is to add some accounts to your database and test it!

To insert a domain into the database connect with mysql -u mail -p mail and enter:
INSERT INTO domain (domain) VALUES ('domain.tld');

To add a mailbox (an actual inbox on the computer) issue the following query in MySQL:
INSERT INTO mailbox (username, password, maildir, domain) VALUES ('user@domain.tld', md5('password'), 'domain.tld/user/', 'domain.tld');

Additionally I always add an alias for users, even if it points to the same address. In the alias table you can create aliases that points to an external mail, for example user2@domain.tld can be set to point to mymail@gmail.com. You can set up an alias to send to several mail addresses by adding several aliases that are the same but points to different targets. To create a catch-all (i.e. where any mails sent to an address of your domain that doesn't have an alias already ends up) enter the alias as @domain.tld. Here are some examples:
INSERT INTO alias (alias, target, domain) VALUES ('user@domain.tld', 'user@domain.tld', 'domain.tld');
INSERT INTO alias (alias, target, domain) VALUES ('user2@domain.tld', 'mymail@gmail.com', 'domain.tld');
INSERT INTO alias (alias, target, domain) VALUES ('user2@domain.tld', 'user2@domain.tld', 'domain.tld');
INSERT INTO alias (alias, target, domain) VALUES ('@domain.tld', 'catch-all@domain.tld', 'domain.tld');

To make the administering of this easier I put together my own administration interface in PHP in which you can easily add, edit and delete domains, aliases and mailboxes using the database I've posted here. It doesn't have any error-checking or security, but it does it's job trusting the user not to screw things up. It's probably a pretty good idea to password-protect it though, if you intend to have it on a server public to the Internet. .htaccess and .htpasswd will do the job just fine, I guess, so Google that.
Here is a download link: nmailadm-0.1.tar.gz

Testing

Now that you've got your mail server set up, it's about time you try sending and reading some mails using Telnet. If it works, you can try using a mail client (such as using POP to fetch mails with Gmail or Outlook to both send and recieve mails).

The easiest way to test is to use telnet. Here's how to send a mail using SMTP:
telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 server.domain.tld ESMTP Postfix

mail from: user@domain.tld
250 2.1.0 Ok
rcpt to: user2@domain.tld
250 2.1.5 Ok
data
354 End data with .
This is a test mail. :)
.
250 2.0.0 Ok: queued as **********
quit
221 2.0.0 Bye
Connection closed by foreign host.


Now, check your maillog and you should see a couple of lines following you through the process you just did. Here is an example:
Sep 13 17:20:27 server postfix/smtpd[11333]: connect from localhost[127.0.0.1]
Sep 13 17:21:33 server postfix/smtpd[11333]: **********: client=localhost[127.0.0.1]
Sep 13 17:22:31 server postfix/cleanup[12236]: **********: message-id=<20090913152133.**********@server.domain.tld>
Sep 13 17:22:31 server postfix/qmgr[30203]: **********: from=, size=355, nrcpt=2 (queue active)
Sep 13 17:22:31 server postfix/virtual[13036]: **********: to=, relay=virtual, delay=81, delays=81/0.01/0/0.02, dsn=2.0.0, status=sent (delivered to maildir)
Sep 13 17:23:22 server postfix/smtpd[11333]: disconnect from localhost[127.0.0.1]


Now that we've sent a message, let's try connecting with POP to list the messages we have!
telnet localhost 110
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
+OK Dovecot ready.

user user@domain.tld
+OK
pass password
+OK Logged in.
list
+OK 1 messages.
1 450
.

quit
+OK Logging out.
Connection closed by foreign host.


Yep, that's about it. If you followed this and everything worked you should have a fully functional mail server with virtual users running! Next is to connect using a mail client such as Outlook so you won't have to send mails using telnet.

If you have any questions or problems, feel free to send me an e-mail or post a comment and I will do my best to help. :)

Friday, September 11, 2009

Google Chrome Extensions Usage Guide

There are probably a lot of people who can't live without their extensions for Firefox, and missing out on the by far (in my opinion) best browser currently available. Google Chrome. It was a problem for me too, until I found out by accident that the developer channel of Google Chrome actually supported extensions, though they were incomplete and even required a special flag to be set when running the Chrome executable. Even then, I felt I had to check this exciting feature out, and boy was I surprised when I found out how easy they are to make and use and how well they work! I quickly made a few quick extensions using most of the techniques available. The most popular is the PageRank Display extension, the first one I wrote, which has users all over the world and linkbacks from all the way over in Japan and Korea.

Even though there are a few online communities about Chrome extensions a lot of people are probably missing out on extensions, first of all because they're not enabled at all in the stable channel and second because you previously needed a special flag in the shortcut even on the dev channel. If you haven't already I strongly urge any Chrome users to switch to the dev channel, it is much further along the development and though it says it can be very unstable at times is has been extremely rare for me. It can be done here: switch between stable/beta/dev channel. The flag in the shortcut isn't a problem once you have the dev channel anymore, last week when they finally made it the default to have extensions enabled in the dev channel (along with some UI changes I've been looking forward to).

Now, how do you use extensions in Chrome? It's really very simple. All you have to do is find a link to an extension and click on it (or if you have an extension file (.crx), you can just drag it to your Chrome shortcut). After accepting the download Chrome will pop up a message asking you if you want to install the extension, which of course you do (unless you distrust the source of course). After that the extension is installed instantly (no need to restart Chrome like you do with Firefox) and you should see any UI the extension adds. Regarding UI, there are four types of them:
Toolstrip:
The toolstrip is a bar at the bottom of your browser window. It automatically allocates enough space to fit the extension since Chrome values the screen real estate of their users, something I greatly appreciate. In the toolstrip there can be basically anything, buttons, links, information, checkboxes, etc. The toolstrip can be hidden with the shortcut ctrl+alt+b. It's the most common form of extension UI.
Moles:
Moles are pretty much an extension of the toolstrip. When you hover over an extensions allocated space in the toolstrip the name comes up. When you click the name you can open a mole (if the extension has created one) which is kind of like a popup page that shown up above the toolstrip. You can, for example, use the toolstrip to display some information and a form with checkboxes etc. in the mole for setting up what and how to display things.
Content scripts:
Content scripts are pretty much like user scripts (i.e. Greasemonkey, javascript files run on selected pages, Chrome has regular user script compatibility too, where you download a script and put it in a folder) in the form of an extension. Besides the possibility to combine other extension features with user scripts (called content scripts) it also gives auto-update and easier installation etc.
Page actions:
Page actions are used when you want the option to do things with a certain page. What it does is give you an icon in the omnibox (address bar) for the extension on the pages where it can be run. When you click on the icon the extension does something with the page. For example the extension can detect when a page has an RSS feed available and enable the page action for the page. An RSS icon will show up in the address bar which will e.g. add the feed to your Google Reader account when clicked.

An extension can combine any number of them in any combination and can communicate between them (as long as they belong to the same extension).

Once you've got the extensions there might be some you want to remove again, or you might simply want to see what extensions you have installed, versions, etc. To do all of this all you have to do is browse to chrome://extensions. Once there you can remove, add extensions from file, add unpacked extensions (they're packed in .crx files, before that they are folders with html and javascript files in them) and force check for updates. You will see a list of all installed extensions as well as their version and a description.

To test how extensions work I suggest you to visit the official sample extensions. As of now there are three sample extensions there:

Gmail Checker


Displays the number of unread mail in the toolstrip.
Gmail Checker screenshot

Subscribe in Feed Reader


Uses page actions to subscribe to a feed.
Subscribe in Feed Reader screenshot

BuildBot Monitor


Shows the status of the Chromium Build Bot (probably not interesting to a regular user).
BuildBot Monitor screenshot

These are two simple extensions that I wrote for some basic SEO work:

PageRank Display


Shows the PageRank of the open site in the toolstrip.
PageRank Display screenshot

Alexa Rank Display


Shows the Alexa Rank of the open site in the toolstrip.
Alexa Rank Display screenshot

To find more Google Chrome extensions I suggest you visit Chrome Plugins. They've got discussions and a lot of extensions posted by users, for example for mouse gestures, translation, smooth scrolling, session saving, weather, alarm clock, video downloader, etc.

I hope this'll get anyone still using other browsers because they've got extensions they need to switch to Chrome - it's got a fair number of great extensions already, and an excellent and easy-to-use extensions API for creating new extensions in case there doesn't yet exist one with the functionality you require. :)