Roxen Community: Session Variables Revisited

Author: Martin Nilsson <nilsson@roxen.com>
Last modified: 2001-09-30 15:59:14
A year ago Kai Voigt made an academic report on web sessions, which led up to his session module 123session as well as an appreciated presentation on the Roxen User Conference and a Roxen Community article. Though one might think that would be quite enough, there is still a lot more that can be said on the topic. In Roxen WebServer a lot of Kais work has been integrated as integral parts of Roxen core, and with this article I hope to widen the scope of the previous article as well as providing in-depth knowledge of how the new primitives can be used both in RXML and Pike.

Session Definition

It is always good to make a formal or semi-formal definition of things before you get your hands dirty, but in this case it is not easily done. Not only are sessions in themselves tricky, but many people have their own, private view of what a session is and those views are not always compatible with each other. Given the assumption that a session has a beginning and an end and has a state that should be taken, modified or not, from the beginning to the end, I'll take a closer look at different uses of "sessions" and try to find the common properties and notable differences between them. Buckle up and be prepared to disagree.

The author, Martin Nilsson
<nilsson@roxen.com>

In a web application, almost by definition, the content of the present page is dependent of user actions on previous pages. It could be as trivial as a search engine result page being dependent of the user provided search query, e.g. one component that depends on a parameter. It could also be more advanced like several independent components, each depending on their own parameters, e.g. several tables, each sorted according to selected columns.

Is this a session? It probably is, as it has a well defined beginning, when the user enters the first page of the web application, and it has a defined, although not very predictable, ending when the user at any point leaves the web application. It should also be noted that one user might concurrently be at different states in different sessions in different browser windows. Thus it is common that the state is identified by the URL or by the URL and form variables. Note that prestates, different domains etc. are all part of the URL. Further it is common that not only is the state identified by { URL, variable } but also completely described by this key, i.e. it is possible that the full state is stored in a database on the server side and only referenced by a key in the { URL, variable } state. The two "drawbacks" are that the burden on the server side increases, since the full state must be looked up at every request, and that the value must be removed by GC at some time.

This type of session is independent of who is browsing and which computer is used. So in lack of a better name, I call it a "Page Session".

Marketing people often visit us developer and ask questions like "How many unique visitors do we have?" and "For how long does a visitor look a the front page?". The answer is of course "It is not possible to know" and the answer to that is usually "Yea, right. Come on, how long?". This is usually followed by the answer "I can give you an approximation." after which both the developer and the marketing guy feels superior.

The core problem here is of course to identify a unique visitor with as good accuracy as possible. Some properties of an incoming request is trivially non-unique, like the user agent (UA) string of the browser. Others are mistakenly considered unique by some, like the IP number of the client. It is more and more common practice to put several computers behind a gateway computer of some kind, e.g. a firewall. It is of course possible to combine the identifiers, e.g. by using { IP, UA } as the identifier, but still a long shot from uniquity.

This is where browser cookies enter the picture. With browser cookies you can set a variable in the browser that the browser then will send during all successive requests, until the cookie expires. There are of course people who think they are so important that anyone would care about how they, as individuals, act on the Internet, and thus turn their cookies off. The key derived from { IP, UA, CookieID } is however often a sufficiently good approximation of browser identification to be used to ask both the questions mentioned above.

The real purpose of identifying a specific browser is to be able to record the behaviour of it as good as possible, for log statistics. The actual session begins when the browser enters the site and ends when it exits the site. Unfortunately, from a statistics point of view, there is no way to record that exit event, so a timeout is used for this. Sometimes this timeout is defined as n times the average distance in time between two successive page requests from the same browser. So, is this a session? Yes, it is a session, since it maps directly to what people perceive as a session on a web site. As so often is the case, what humans perceive as the best example for a certain phenomenon turns out to be technically very hard to measure. On the upside, we get a fairly good identification of the browser that can be used to answer the question about unique visitors, which has nothing to do with sessions.

This type of session is independent of the actual pages viewed, and any concurrency that might occur, e.g. different parts of the site viewed at the same time in different windows in the same browser. Neither does it have anything to do with the actual user that is browsing the site. The same user can browse the site daily with his dial up account, and still be recorded as a different user most of the times. Thus I call this session type for "Browser session".

When using a shopping site it is convenient to have a virtual shopping basket. In it you can place items you intend to purchase, and at any time you can select "checkout" and have its contents delivered to your preferred shipping address. Nice, hype and future. Preferably this shopping basket isn't confined to a page session, since you might want to call up several products in adjacent windows and compare them, and then you would like to be able to put them in the same shopping basket no matter in which window an item is selected. Furthermore we don't want to lock the shopping basket to a particular browser either. We might select a few items, go to work, select a few more and then commit a checkout.

Is this a session? It certainly is. There is a well defined beginning when we enter the shop and begin selecting items. The state is preserved throughout all pages in the shop, and preferably throughout all browsers we use, until we commit a checkout, which is a well defined end of a shopping session. There is no defined upper time limit on this session, and most online shopping sites probably feel reluctant to GC shopping sessions in progress. Given the big scope of these sessions we can not use URL or form variable based session, nor is cookies sufficient. The unique identifier must be a user ID, which given the current shortcomings of technological advances in digital identification must be provided by the user manually. Since entering it during every page load is far from convenient, it is practical to connect a user ID to a browser session, preferrably a cookie.

This is of course a "User session", since it is independent of browsers.

Creating an ID

Looking at my (perhaps incomplete) list of different types of sessions above I very intentionally went from a very local use, preserving states at specific pages, up to the global level of recognizing a specific user and use that users settings/state on the site. Surprisingly, the difficulties involved are quite the opposite. It is fairly simple to design a mechanism that maps a user, through the use of global variables such as cookies, to a certain internal state. It is much more complicated to let discrete and independent components transport a state to another page, without altering the other components state. At least if it is to be done in a general way.

Thus we begin with looking at user sessions. Essentially there is nothing more to it than creating a database where each row gets to represent a user and its state. When the user logs in, she is verified against some property in her user state, e.g. a password. Once verified, a random value is generated and stored both in a user cookie and as a variable in the user state. For as long as the value in the user cookie is the same as in the database, the user is considered being logged in. Thus if the user logs in at another place, the old value is no longer valid and that session is automatically logged out. This last feature is not something that is required for a user session; quite the opposite, but it adds an often wanted layer of security. We are also able to create a 100% working log out button that sets the value in the database to something that isn't a valid random value, e.g. the database NULL value. The concept of log out is not present in the HTTP protocol, since it is stateless. When a user logs in with HTTP authentication, the stated "logged in" is kept by the browser, which sends the validated username and password in every successive request.

From a programming perspective, be it Pike or RXML, there is only one problem left to solve; how to create a good random value. The method presented by Kai Voigt in "Session Variables with Roxen" is implemented, in a slightly different version, in Roxen WebServer core from version 2.2. It currently looks like this
private int unique_id_counter;
string create_unique_id()
{
  object md5 = Crypto.md5();
  md5->update(query("server_salt") + (unique_id_counter++) + time(1));
  return Crypto.string_to_hex(md5->digest());
}
It essentially does the same thing as Kais sessionid_create, but it uses a simple local integer, that will be resett to 0 at every restart, except for using one that is always greater than before. This prevents a disk write between every request. The only risk of getting a non-unique id is if the server restarts and manages to do so under one second. If anyone has this problem, contact me...

For pike programmers this method is available as roxen.create_unique_id and for RXML programmers as &roxen.unique-id;. Note that the entity generates a new entity every time it is used. If you need the same value several times in your code, first copy its value into a normal variable.
  <set variable="var.id" from="roxen.unique-id"/>
Setting an ID

The world is of course not as idealized as my categorization above. Most often a browser session used as a user session will be enough, thus saving the hassle of user management. Just the expressions "privacy concerns" and "lost password" is enough for most to settle for this solution. Furthermore it is often a requirement that one and only one browser gets into a certain session. As discussed above we can not be sure that cookies are turned on, so if it is more important that the session is unique than browser global, we can downgrade to a page session, if required. A possible session imposing protocol would thus be:

Is there a session cookie present? If so we have identified our session and are done.

Send a cookie and redirect to another page.

Is there a session cookie present, i.e. did the client accept the cookie we tried to se in the last step? If so we created a new session ID and are done.

Redirect to another page where the session ID is present in the URL.

We have created a new session ID and are done. The session will however be confined to the present browser window.

In order to automatically have every browser tagged with a unique ID, as discussed in the browser session section, change the "Set unique user id cookies" to "Yes" under the ports tab in the Administration Interface. In WebServer 2.2 the cookie RoxenUserID will be set to a value acquired from the create_unique_id function. You can combine this setting with using to <forxe-session-id/> tag to create a behaviour similar to the above, but somewhat compressed in its actions. The following is executed every time someone looks at the page.

If there is no RoxenUserID cookie present AND no RoxenUserID prestate present then redirect to the same page, but with a prestate like RoxenUserID=X set, where X is an ID generated as above.

If there is a RoxenUserID cookie AND a RoxenUserID prestate present then redirect to the same page, but with the RoxenUserID prestate removed.

It is important though to realize that just because we have defined a browser session ID that doesn't mean we neither with necssity interfere with nor change interest in page session ID:s or user session ID:s. As we have seen a user session can be mapped onto browser sessions by connecting a browser session ID in all browsers to the user session ID. In the same way can a browser session be mapped onto page sessions simply by connecting the browser session ID to all page sessions. In theory. Due to technicalities we can generally not alter the page state of other browser windows, hence we map the browser session onto a page session in a single window as a best effort solution.

ID - Variable Connection

So far we have not done much in the way of connecting real data to our session id:s. When dealing with visitor statistics, there is no additional data to be connected to the id. When dealing with user sessions as above, there is already a database in place with a row for each user, giving a natural place for any session bound information to reside. These two examples are however not representative for sessions. Usually there is a session, a session id and one or more session variables. Once we have succeeded in defining the session and connecting the session id to it in a server readable way there is no big problem in creating an id variable connection. Just consider the variable storage as a cache (read my cache article for more information) with different demands on cache integrity and GC policies.

To biggest difference is that we can not throw out entries arbitrary, since they can not be recreated. The GC policy must therefore only remove entries that has expired, a condition normally met only if the entry has been unused for a certain amount of time or if the entry has been explicitly expired by a user action. In a normal cache system it is also possible to use properties like entry size and entry generation time, which is not possible here.

There is also a higher demand on the integrity of the session variable storage, than on a cache. Again, here the values can not be automatically regenerated if e.g. the server is restarted, so session values should be stored in a database. This may also be part of the GC policy, to store entries in a database after a certain time of non-use. This time factor may however successfully be dependent of the entry size. Note that even if the session variables are stored in a database right away, there is nothing that stops us from having a cache in front of the database when reading data.

A cache/database system design as above is available in the Roxen WebServer cache system, through the three methods cache.set_session_data, cache.get_session_data and cache.clear_session.

string set_session_data(mixed data, void|string id, void|int persistence, void|int(0..1) store)
Associates the session id to the data. If no id is provided a unique id will be generated. The session id is returned from the function. The minimum guaranteed storage time may be set with the persistence argument. Note that this is a time stamp, not a time out. If store is set, the data will be stored in a database directly, and not when the garbage collect tries to delete the data. This will ensure that the data is kept safe in case the server restarts before the next GC. Also not that the provided data must not contain any object, programs or functions, or the storage in database will throw an error.

mixed get_session_data(string id)
Returns the data associated with the session id. Returns a zero type upon failure.

void clear_session(string id)
Removes the session data associate with id from the session cache and session database.

These functions work against a cache/database that consists of four buckets which are shifted every fifteen minutes. These properties may be altered by setting the defines SESSION_BUCKETS (which defaults to 4) and SESSION_SHIFT_TIME (which defaults to 900) when the server is started. When a session value is fetched it will be moved to the first bucket. If a value in the out-shifted bucket hasn't expired it will be stored in the session database. Note that no entry is removed before it has been shifted through all buckets, no matter how early the persistence value has been set.

This functionality is also exported to RXML through the Session tag module. By using the container tag <session> you can create a new scope which accesses the session variables associated with the id given in the attribute id to the tag. Combining id creation, setting and session gives you the following example.
<force-session-id/>
<if variable="client.session">
  <session id="&client.session;"><pre>
    <form action="&page.self;" method="post">
    Session value: &_.val;
    Form value: &form.val;
    Update value: <input name="val" value="&_.val;" />
    <input type="submit" value="Update" />
    </form>
    <if variable="form.val"><set variable="val" from="form.val" /></if>
  </pre></session>
</if>
<else>
  <h3>Unable to set a session to your client.
      Neither cookies nor redirects worked.</h3>
</else>
This example first forces a session id to the client. Once the session id is created and set, the contents of the if-statement will be run at every page view. Inside the if-statement we first create a session scope, identified with the session id. In it we print out the current value of the session variable "val" as well as the current value of the form variable "val". We also print out a form widget to enable easy submission of new values of the form value "val". It is of course possible to set the value manually by adding "?val=new" to the URL, where "new" is the new value of "val".

After the HTML form is a line of RXML that sets the local variable "val" (i.e. "_.val" or "session.val") to the same value as the form variable "val" (i.e. "form.val"), if the form variable is set at all. The combined effect should be that the printed session value always is the last form value. It should not matter if you close your browser entirely and then revisit it within an hour.

It is also possible to clear a session scope from RXML by using the <clear-session/> tag.

Componetized pages

The concept of page sessions are easy to grasp and a basic implementation is very easy to accomplish. A session variable can be kept in the URL, normally as a form variable, and then added as a hidden variable in all HTML forms and to all links that points to other pages in the realm of the page session. Sounds fuzzy? Consider the following example, a web based adventure game. Let's say we mimic one of the adventure game books where in the end of each section you get to choose what to do next. Beside each choice was a section number to which one should jump to continue the adventure with the selected gameplay choice. Imagine these sections as different web pages. To every gameplay session we would like to attach e.g. an inventory and some character parameters like health, skills and strength, which we can accomplish by using a session.

The session should preferably be confined to the window in which the game is played (ok, this enables the user to play different scenarios in windows side by side, but that was more or less possible in the books as well, by using a bookmark). We have two different choices regarding session identification. Either we add a session id to all links, so that we can keep track of which session is connected to the current game, or we add the variables themselves to each link. The advantages of the later method are many. We don't need to keep any session variables on the server side, thus removing the need of GC. This will ensure that the session variables will always be present, no matter how much the user waits between page loads. The pages will also behave somewhat more predictable from a user point of view, since history operations (back, forward etc.) and bookmarks will work. These reasons makes this way of handling page sessions the far most used of all.

Though the page sessions are easy to handle on a technically low level, e.g. how to actually transmit the session variables/id to another page, things get much hairier when you make any attempt to create a model for componentized web pages, pages like the one built with RXML. Let me give you a simple, hands on example. We want to list some items from a database, and we want it to be possible to sort them in different ways.
<define tag="list">

  <if variable="form.order = ASC">
    <a href="&page.self;&amp;order=DESC">Descending order</a><br />
  </if><else>
    <set variable="form.order">DESC</set>
    <a hef="&page.self;&amp;order=ASC">Ascending order</a><br />
  </if>

  <emit source="sql" query="SELECT name FROM &_.table; ORDER BY NAME &form.order;">
    &sql.name;<br />
  </emit>

</define>

<list table="friends" />
Some explanation of the above code. We define a tag of our own with the name "list". In it we first check if the form variable "order" has the value "ASC". If so we present a link pointing to the same page, but with the order variables set to the value "DESC" instead. In the if tags else branch we first set the form variable order to "DESC", since if this is the first page view it will not have any (or wrong) value. Then we present a similar link as in the if main branch, but we set the value to "ASC" instead. Finally we use an emit to output all the values of the name column in the SQL table, whose name is given in the list tags table attribute, and sort that list according to the order form variable. In the end of the example we use the list component. So far this will not cause any problems, but consider we want several of these "list" components on the same page.
<p><list table="friends" /></p>
<p><list table="enemies" /></p>
There will now be interference between the both list components, since both of them are using the same variable to store their own state. The problem doesn't go away if we use different components either, since altering the state in one component will then reset the state in all the other components. What is needed is a central manager of all components states that is aware of everything that has been changed, and generates that appropriate state that needs to be transmitted to the next page. This need was answered already in Roxen WebServer 2.0, due to problems with coexisting foldlist tags (or flik tags), and the state handler was introduced. This has not really been mentioned before, since the state handler is not that simple to use.

State handler

The state handler, available from Pike in WebServer as StateHandler, registers different page components under different names and keeps track of their states. The first thing a (tag) module has to do after it has created a state object, is to register itself in the page state object. This is done by providing a suggested id, typically the name of the tag. The registration method then returns the given id, which may be a different one than the suggested id.
  string state_id = "my-tag";
  object state = Page_state(id);
  state_id = state->register_consumer(state_id, id);
Then it is a good idea to update the state object with the current page state, transported from the last page. The state handler doesn't care about how the state is transferred between pages, but an convention among Roxen-made RXML tags is to use the __state form variable.
  if(id->real_variables->__state &&
     !state->uri_decode(id->real_variables->__state[0]))
   RXML.run_error("Error in state.\n");
It is now possible to retrieve the state associated with your page component by calling the get method in the state object. It is however possible that the state was already available before this registration, since there might have been an earlier component on the page that already had the state registered.

When dealing with page sessions it is important to understand that the page state is changed "between" pages. Consider a page with two components, the first uses the form variable a and the second the form variable b. If we are looking at the page page.html?a=1&b=1 the page source might look like
Component A<br />
<a href="page.html?a=1&b=1">1</a>
<a href="page.html?a=2&b=1">2</a>

Component B<br />
<a href="page.html?a=1&b=1">1</a>
<a href="page.html?a=1&b=2">2</a>
Note that the links contains the state of the page it links to. Hence, during the generation of this page all the next following possible states must be calculated. By this reason it is unlikely that the state in the state object is actually changed during parsing of a page. What is wanted is an encoding of the current state plus a little change, i.e. the change that brings us to the next state. If we continue our state-using module the components above would be programmed as follows:
  string get_actions(string uri, int current_state,
                     object state) {
    return "<a href='" + uri + "?__state=" +
           state->uri_encode(1) + "'>1</a><br />"
           "<a href='" + uri + "?__state=" +
           state->uri_encode(2) + "'>2</a>";
  }
There is still more to be said about sessions, but I think this is enough. My intention was only to present the new session features in Roxen WebServer 2.2, but the article grow as I went along writing it. Sorry, my fault.