A question I hear quite frequently is, how do you match a number within a range in a regular expression? This is a pretty odd question, since regular expressions tend to deal with strings without worrying about the meaning of their content. Most of the time it turns out the person asking is trying to do all of their input validation using regular expressions, which I believe is a singularly bad idea.

Here's an example of what I'm talking about, to match an integer between 0 and 255 (like an octet of an IP address) we could use the following pattern:

25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[0-9][0-9]?

For the pragmatists out there, one arguably valid use case is searching log files for lines about a particular subnet. Anyway, I think that this is an interesting enough problem to tackle without worrying much about practical applications.

About a year ago I came across Justin Rogers' solution to this very problem in C#. As much as I admire his coding skills, his solution seemed to me rather verbose and heavy with special cases. I thought I could do better, or at least understand better how he ended up with that solution by tackling it myself.

I started by writing around 20 test cases, then porting his solution to PHP, so I would have a reference implementation to check my tests against. It took me a while to come up with my own working version, but I was happy to see it passing the carefully crafted tests. I was even happier that it was a fraction of the size of the reference implementation with no special cases to mar its elegance.

However I had a nasty surprise after adding a couple more tests, my solution mysteriously failed them. Evidently my hand written test cases (as usual, some random and some boundary counditions) weren't comprehensive enough. It took me quite a while to find general fixes, but then my happiness returned for a brief visit.

To quell a worry, I decided to generate a hundred more random test cases, AKA. fuzzing. Inevitably more failures emerged, about ten of the new tests failed.

With a decent number of test runs to study, it was quickly apparent that the major issue lay with ranges of the same length which shared a common prefix, e.g. 223-229. My limited test cases from before had led me in the wrong direction.

Eventually I had one failing test out of about 120, which illuminated a subtle error in the logic. After fixing that I ran the code against 50,000 more randomly generated test cases, and was finally assured that it was working.

The code below, I can quite confidently say, works fine. If you find a bug in it then please send me a self-addressed machete so I can thank you properly.

<?php
function generateRange($min, $max)
{
    $digit = '[0-9]';
    $lenMax = strlen($max);
    $lenMin = strlen($min);
    $lenDiff = $lenMax - $lenMin;
    $min = str_pad($min, $lenMax, 0, STR_PAD_LEFT);
    $max = (string)$max;
 
    // find length of common prefix
    for ($i = 0; $i < $lenMin && $min[$i] == $max[$i]; $i++);
    $prefixLength = $i;
 
    // add non-conflicting ranges from each end
    for ($i = $lenMax, $j = 0; $i-- > 1 + $prefixLength; $j++) {
        $lower = $min[$i];
        $upper = $max[$i];
 
        // correct bounds if not final range
        if ($j) {
            ++$lower;
            --$upper;
        }
 
        // lower bound
        if ($lower < 10) {
            $char = $lower == 9 ? 9 : '[' . $lower . '-9]';
            $pattern[] =
                ($j >= $lenMin ? '' : substr($min, $lenDiff, $i - $lenDiff))
                . $char . str_repeat($digit, $j);
        }
 
        // upper bound
        if ($upper >= 0) {
            $char = $upper ? '[0-' . $upper . ']' : 0;
            $pattern[] = substr($max, 0, $i) . $char . str_repeat($digit, $j);
        }
    }
 
    // add middle range
    if (!$j || $max[$prefixLength] - $min[$prefixLength] > 1) {
        $prefix = substr($min, 0, $prefixLength);
        $lower = $min[$prefixLength];
        $upper = $max[$prefixLength];
 
        // correct bounds if not final range
        if ($j && $i == $prefixLength) {
            ++$lower;
            --$upper;
        }
 
        $char = $lower == $upper ? $lower : '[' . $lower . '-' . $upper . ']';
        $pattern[] = $prefix . $char . str_repeat($digit, $lenMax - $prefixLength - 1);
    }
 
    return join('|', $pattern);
}
?>

My solution ended up along the same lines as Justin's, and although I had achieved my original aim of understanding his solution better, the real educational experience here was the inadequacy of my initial tests. I think I've now thoroughly learnt the lesson of testing with a comprehensive range of input.

Bearing in mind that this is just one function with a relatively small domain of useful input: If you fuzzed your own applications, what obscure bugs might you find?

As an aside, when I came back to this recently having decided to write an article on the topic, I found an elegant solution, written in Python (of course), which uses the rather different approach of first partitioning the ranges with a recursive function and then constructing a regular expression for each one.

Out of curiosity I ported this to PHP too, but as you might expect it became decidedly less elegant and the depth of recursion segfaulted PHP on all but the most trivial input.

Photo by Jag arts.

One of my geekiest hobbies is playing golf. Not the 18 holes of weathered drudgery kind, but the 18 hours of staring at the same 2 lines of code kind. If you still don't know what I'm talking about, the aim of the game is solve a given problem in the smallest amount of code (fewest strokes... geddit?).

Code golf has been played with various languages since at least the '50s, according to the famous Perlgolf History book, but it was certainly the Perl golfers who took it to the next level. One might say Perl is perfectly suited for golf because of its perversely rich and succinct syntax, but I think golf's popularity in the Perl world says more about the mentality of the people who inhabit it. I can't imagine many VB.NET coders getting into it, even if their language wasn't so verbose.

Two years ago, 29 degrees launched a brand new golfing website, codegolf.com. The slick interface was a major attraction, no fiddling with test files required, just submit your code and see if it works. However, codegolf.com's biggest feature is that Perl, PHP, Ruby and Python can all compete on the same challenges together. While your PHP entries are unlikely to be troubling the top Perl entries, there's a fairly good chance of embarrassing some Ruby and Python coders

Since the PHP world is pretty new to all this, I've assembled some tips and tricks you can use to shorten your PHP solutions according to the codegolf.com rules. Hopefully needless to say, these should never get near any kind of production code.

• Get a working version first

  This might seem like stating the obvious, but it's easy to miss a subtle error in a test case and then waste time golfing you code you have to throw away.

• Notices? What notices?

  Notices will be ignored, so you can skip setting variables to 0, null, "" or array() at the beginning of the script. This also means you can drop the quotes from some strings.

• The function with the shortest name may not be the shortest

  Don't always fixate on the functions with the shortest names, quite often a function with a longer name will save work elsewhere.

• Lookup tables

  You can often cut out a large part of code by looking the result up from a string rather than calculating it. Consider using pack() to squeeze more data into the same string. Aim to produce an alphanumeric string, so you can drop the quotes.

• Looping

  Foreach loops are rarely the shortest. While and for loops are the same length, but for gives you two free semi-colons so it usually wins. Always try keep loops to one statement, so you can drop the braces.

• There's always another algorithm

  If you've spent ages on one approach and are still 50 bytes behind the guy ahead, it's probably time to think of a new algorithm.

• Bitwise tricks

  Bitwise OR with 0 is a nice alternative to floor(), so floor($a / $b) becomes $a / $b | 0. Changing the case of a letter in a string is another good one - $c & "\xDF" is the same is strtoupper($c), and $c | " " is the same as strtolower($c).

• Use literal characters instead of escapes

  While golfing it's easiest to keep characters escaped, but once you're ready to submit you can often save an extra few bytes by replacing escapes such as \n with a literal newline.

Beware, code golf is highly addictive and can lead to broken keyboards, windows and families.

<?php
$str = 'unquoted section "quoted" and unquoted
    "another quoted \"with a subquote\" section" more unquoted';
preg_match_all('/"(?:\\\\.|[^"])*"|[^"]+/', $str, $m);
print_r($m);
?>

Output

Array
(
    [0] => Array
        (
            [0] => unquoted section
            [1] => "quoted"
            [2] =>  and unquoted
            [3] => "another quoted \"with a subquote\" section"
            [4] =>  more unquoted
        )

)

<?php
$a = array('a', 'b', 'c');
foreach (array_reverse($a) as $v) {
    $b = isset($b) ? array($v => $b) : array($v);
}
print_r($b);
?>

Output

Array
(
    [a] => Array
        (
            [b] => Array
                (
                    [0] => c
                )

        )

)

Like register_globals, it helped lower the barrier of entry to building a dynamic website by removing some of the complexity. However it certainly wasn't without sacrifice, problems with the implementation quickly appeared and continued to abound for the next ten years. Finally in PHP 5.2.2 we got an implementation which (as far as its intentions went) seemed to be bug free, but of course by then it was turned off by default and was already slated to be dropped in PHP 6.

Why are magic quotes so bad?

• Insufficiency

  This is the single biggest problem with magic quotes. Even if you follow all the instructions right and code defensively, relying on magic quotes to escape strings for database queries is still not secure. Magic quotes uses addslashes() for its escaping which doesn't care about the character encoding of your database, meaning it's vulnerable to multi-byte character attacks as explained by Chris Shiflett.

• Portability

  You simply cannot rely on magic quotes being on or off if you expect to distribute your PHP script to others, since many hosts don't provide a way of changing the setting.

• Complacency
  

  The very concept of magic quotes fosters complacency. Why bother validating input if it's going to be escaped anyway? All too often on IRC I see code which completely neglects validation - strings and supposedly numeric values going straight into the query with no effort being made to check them. Examples can be found in countless books and tutorials, and they are still being written.

• Inconsistency
  

  Depending on which version of PHP you use, you can expect vastly different results from magic quotes. As mentioned above, PHP 5.2.2 was the first version in which magic quotes actually did what it said on the tin. Some features which may or may not work include: escaping double quotes; escaping the $_SERVER and $_ENV arrays; escaping the keys of arrays.

• Performance
  

  When magic quotes is on, everything is escaped. You might have a page which only uses PHP to output the current date, but if the user decides to send 8 MB of post data, it's going to be escaped whether you like it or not, before your script even starts. Extreme examples aside, any posted value which you wouldn't need to escape is another useless call to addslashes().

• Inconvenience
  

  As convenient as it is not worrying about having to escape anything, once you're aware of magic quotes' flaws, they become a real pain in the arse. We've all seen sites suffering from double escaping, where content ends up being actually output like "the Pope's underpants".

Coping with magic quotes

The number one piece of advice regarding magic quotes has to be, turn them off! However that's easier said than done. Since it takes effect before the script is run you can't just use ini_set() at the top of your script.

The luckiest people host their own sites or have a host agreeable enough to turn it off in php.ini or the web server configuration. The slightly less lucky people can still control PHP settings per-directory, for example by the use of an .htaccess file.

Everyone else is forced to try to cope with it at runtime, and this is where the real fun begins. Since PHP 4, there's been the handy get_magic_quotes_gpc() function, which is most commonly seen used like the following example from the PHP manual:

<?php
if (!get_magic_quotes_gpc()) {
    $lastname = addslashes($_POST['lastname']);
} else {
    $lastname = $_POST['lastname'];
}
?>

Unfortunately, get_magic_quotes_gpc() is not aware of the magic_quotes_sybase setting, which changes the escape method to Sybase-style doubled quotes, and also knows nothing of the various inconsistencies of magic quotes. Therefore you could easily be applying addslashes() where you don't need to and vice-versa.

The other common approach is to try to completely undo the effects of magic quotes, in the hope that it will end up as if magic quotes was never turned on, and then proceed with proper escaping. This solution means that at worst you'll be double escaping - there's no chance of anything dodgy getting through as long as your proper escaping is effective. However, it has some problems of its own.

At the primitive end of things, you might see code like the following:

<?php
if (get_magic_quotes_gpc()) {
    $_GET = array_map('stripslashes', $_GET);
    $_POST = array_map('stripslashes', $_POST);
    $_COOKIE = array_map('stripslashes', $_COOKIE);
    $_REQUEST = array_map('stripslashes', $_REQUEST);
}
?>

Ignoring the potential double-escaping, this solution has two fundamental problems. Firstly any arrays in the input will be converted to the string "Array", which is slightly less useful. Secondly it only fixes the values, ignoring the keys.

It is quite simple to make a recursive function to tackle the first problem, and that's what the PHP manual opts for on its page on Disabling Magic Quotes:

<?php
if (get_magic_quotes_gpc()) {
    function stripslashes_deep($value)
    {
        $value = is_array($value) ?
                    array_map('stripslashes_deep', $value) :
                    stripslashes($value);
 
        return $value;
    }
 
    $_POST = array_map('stripslashes_deep', $_POST);
    $_GET = array_map('stripslashes_deep', $_GET);
    $_COOKIE = array_map('stripslashes_deep', $_COOKIE);
    $_REQUEST = array_map('stripslashes_deep', $_REQUEST);
}
?>

The recursion means that it won't destroy your arrays but it still suffers from the second problem, not escaping the keys, and it introduces yet another problem - arbitrary recursion depth. Consider the following query string:

foo[][][][][][][][][][]=1&foo[][][][][][][][][][1]=2

Fixing those two values alone takes 22 function calls, imagine how much work a single request could cause using the maximum post data size - by default 8 MB. That's a pretty effective basis for a DoS attack.

In practice PHP will fall over far before it completes all that work, namely because it sucks at recursion. A new configuration setting (max_input_nesting_level) was added in 5.2.3 to control the maximum recursion depth, but hosts providing 5.2.3 or above are sadly in the minority.

The easy solution is to add another argument and a depth check to that function, but that leaves the problem of escaping the keys, and the double-escaping problem we've been ignoring from the beginning.

Enter, PHP_Compat. Aidan Lister pioneered this awesome PEAR package and I've proudly been helping out for the last couple of years.

In our upcoming release, PHP_Compat 1.6.0, we've added an environment module which includes magic_quotes_gpc among the settings it fully supports changing at runtime. Because of the nature of the package we've gone with the second approach, trying to restore the environment to exactly how it was before magic_quotes_gpc took effect. However we use an iterative algorithm so recursion depth isn't a problem and we take into account every inconsistency going back to PHP 4.0, so you shouldn't experience any double-escaping at all.

The future

Until they're gone for good, we have to spread the message that magic quotes are evil, turn them off! The less people make use of magic quotes, the sooner we can be rid of them. PHP 6 is continually being delayed and at the rate people have moved to PHP 5 it will take at least another ten years for everyone to be on PHP 6. I'm sure we'll see plenty more adventures along the way.

Update:

For completeness I think I should point out here, that yet another problem with magic_quotes_gpc emerged with PHP 5.2.7, it was completely broken.

Also Chris Shiflett added in a kind comment on phpdeveloper.org:

One thing I would add to your list is the fact that magic quotes is an escaping solution. Escaping input provides a unique set of challenges, because it's an inappropriate design decision. Because data is not necessarily destined for exactly one context, it is only important to be sure input is valid. (Filter input.) Each context can then be handled as needed, typically when the data leaves PHP for another context. (Escape output.)

Magic quotes violates this simple tenet, and in my opinion, this is the root cause of most of the problems related to using it.

<?php
 
/**
 * Generates a random string.
 *
 * @param    int    $length
 *   The length of the string
 * @param    string $ranges
 *   The characters to use, formatted like a regex character class (a-fA-F0-9)
 * @param    bool   $unique
 *   Every character in the result is unique or not
 * @return   string
 *   The resulting string.
 */
require_once('preg_expand.php');
function random_string($length = 6, $ranges = 'a-zA-Z0-9', $unique = false) {
    $out = '';
    $chars = preg_expand($ranges, 0);
    if ($unique) {
        $keys = array_rand($chars, $length);
    } else {
        for ($i = $length; $i--;) {
            $keys[] = array_rand($chars);
        }
    }
    foreach ($keys as $k) {
        $out .= $chars[$k];
    }
    return $out;
}
 
/**
 * Quick and simple version
 */
function random_string_simple($length = 6) {
    $chars = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789';
    for ($i = $length, $out = ''; $i--;) {
        $out .= $chars{rand(0, 61)};
    }
    return $out;
}
 
?>

Example

<?php
 
echo '<h4>Default:</h4>', random_string(), '<br /><br />';
 
echo '<h4>8 characters, A-F0-9:</h4>', random_string(8, 'A-F0-9'), '<br /><br />';
 
echo '<h4>6 characters, a-z, unique:</h4>', random_string(6, 'a-z', 1), '<br /><br />';
 
echo '<h4>Simple:</h4>', random_string_simple();
 
?>

Output

Default:

shAjlw
8 characters, A-F0-9:

9A181CE7
6 characters, a-z, unique:

fgkqsw
Simple:

RsxWZj

<?php
 
/**
 * Expands all characters from the given PCRE character class definition. 
 *
 * @param    string    $in
 *   The input character class (not contained in '[]')
 * @param    bool      $string
 *   Set to true to return the result as a string
 * @param    int       $begin
 *   The start of the character range to consider
 * @param    int       $end
 *   The end of the character range to consider
 * @return   mixed
 *   All the characters specified by the input. This is a string if the $string
 *   argument has been provided, or else an array with one element per character.
 */
function preg_expand($in, $string = true, $begin = 0, $end = 127)
{
    if (!is_string($in) || !($len = strlen($in))) {
        return false;
    }
    $range = range(chr($begin), chr($end));
    $in = strtr($in, array('[' => '\[', ']' => '\]', '/' => '\/'));
    $in = ($in{0} == '^' ? substr($in, 1) : '^' . $in);
    $in = preg_replace('/[' .$in . ']/', '', implode($range));
    return $string ? $in : str_split($in);
}
 
?>

Example:

<?php
 
echo 'Alphabet: ', preg_expand('a-z', 1), '<br /><br />';
 
echo 'Hexadecimal: ', preg_expand('A-F\d', 1), '<br /><br />';
 
?>

Output:

Alphabet: abcdefghijklmnopqrstuvwxyz

Hexadecimal: 0123456789ABCDEF

<?php
/**
 * Turns a number of seconds into a human readable time.
 * This function is NOT calendar aware, months and years are based on the
 * average days per year in the Gregorian calendar.
 *
 * @param  int    $in
 *   the input number of seconds
 * @return string
 *   a string representation of the input, broken down into larger units
 */
function secondsh($in, $limit = 0, $sep = ', ')
{
    $units = array('second', 'minute', 'hour', 'day', 'week', 'month', 'year');
    $factors = array(1, 60, 3600, 86400, 604800, 2629746, 31556952);
    for ($i = 7, $bal = 0; $i--;) {
        $cur = floor(($in - $bal) / $factors[$i]);
        if (!$cur && ($i || $in)) {
            continue;
        }
        $bal += $cur * $factors[$i];
        $ret[] = $cur . ' ' . $units[$i] . ($cur != 1 ? 's' : '');
    }
    if ($limit) {
        $ret = array_slice($ret, 0, $limit);
    }
    return implode($sep, $ret);
}
 
?>

Example

<?php
 
$a = array(0, 1, 30, 60, 3599, 3600, 100000, 2419200, 2500000, 555555555);
 
echo '<strong>Default</strong><table>';
foreach ($a as $v) {
    echo '<tr><td style="padding-right:20px">' . $v . '</td><td>' . secondsh($v) . '</td></tr>';
}
echo '</table>';
 
echo '<br /><br />';
 
echo '<strong>Limit 2</strong><table>';
foreach ($a as $v) {
    echo '<tr><td style="padding-right:20px">' . $v . '</td><td>' . secondsh($v, 2) . '</td></tr>';
}
echo '</table>';
 
?>

Output

Default



0
0 seconds


1
1 second


30
30 seconds


60
1 minute


3599
59 minutes, 59 seconds


3600
1 hour


100000
1 day, 3 hours, 46 minutes, 40 seconds


2419200
4 weeks


2500000
4 weeks, 22 hours, 26 minutes, 40 seconds


555555555
17 years, 7 months, 1 week, 20 hours, 39 minutes, 9 seconds




Limit 2



0
0 seconds


1
1 second


30
30 seconds


60
1 minute


3599
59 minutes, 59 seconds


3600
1 hour


100000
1 day, 3 hours


2419200
4 weeks


2500000
4 weeks, 22 hours


555555555
17 years, 7 months