CHUNGA - Portable chunked streams for Common Lisp

Abstract
Chunga implements streams capable of chunked encoding on demand as defined in RFC 2616. For an example of how these streams can be used see Drakma.
The library needs a Common Lisp implementation that supports Gray streams and relies on David Lichteblau's trivial-gray-streams to offer portability between different Lisps.
Chunga is currently not optimized towards performance - it is rather intended to be easy to use and (if possible) to behave correctly.
The code comes with a BSD-style license so you can basically do with it whatever you want.
Download shortcut: http://weitz.de/files/chunga.tar.gz.

Download and installation
Support and mailing lists
The Chunga dictionary
Acknowledgements

Download and installation

Chunga together with this documentation can be downloaded from http://weitz.de/files/chunga.tar.gz. The current version is 0.2.1.

Chunga depends on the FLEXI-STREAMS library. You can download and install Chunga and its dependencies automatically with ASDF-Install, and there's a port for Gentoo Linux thanks to Matthew Kennedy.

For questions, bug reports, feature requests, improvements, or patches please use the drakma-devel mailing list. If you want to be notified about future releases subscribe to the drakma-announce mailing list. These mailing lists were made available thanks to the services of common-lisp.net.

The Chunga dictionary

Chunked streams

Chunked streams are the core of the Chunga library. You create them using the function MAKE-CHUNKED-STREAM which takes an open binary stream (called the underlying stream) as its single argument. A binary stream in this context means that if it's an input stream, you can apply READ-SEQUENCE to it where the sequence is an array of element type OCTET, and similarly for WRITE-SEQUENCE and output streams. (Note that this specifically holds for bivalent streams like socket streams.)

A chunked stream behaves like an ordinary Lisp stream of element type OCTET with the addition that you can turn chunking on and off for input as well as for output. With chunking turned on, data is read or written according to the definition in RFC 2616.

[Standard class]
chunked-stream

Every chunked stream returned by MAKE-CHUNKED-STREAM is of this type which is a subtype of STREAM.

[Standard class]
chunked-input-stream

A chunked stream is of this type if its underlying stream is an input stream. This is a subtype of CHUNKED-STREAM.

[Standard class]
chunked-output-stream

A chunked stream is of this type if its underlying stream is an output stream. This is a subtype of CHUNKED-STREAM.

[Standard class]
chunked-io-stream

A chunked stream is of this type if it is both a CHUNKED-INPUT-STREAM as well as a CHUNKED-OUTPUT-STREAM.

[Function]
make-chunked-stream stream => chunked-stream

Creates and returns a chunked stream (a stream of type CHUNKED-STREAM) which wraps stream. stream must be an open binary stream.

[Specialized reader]
chunked-stream-stream (stream chunked-stream) => underlying-stream

Returns the underlying stream of the chunked stream stream.

[Generic reader]
chunked-stream-input-chunking-p object => generalized-boolean

Returns a true value if object is of type CHUNKED-INPUT-STREAM and if input chunking is currently enabled.

[Specialized writer]
(setf (chunked-stream-input-chunking-p (stream chunked-input-stream)) new-value)

This function is used to switch input chunking on stream on or off. Note that input chunking will usally be turned off automatically when the last chunk is read.

[Generic reader]
chunked-stream-output-chunking-p object => generalized-boolean

Returns a true value if object is of type CHUNKED-OUTPUT-STREAM and if output chunking is currently enabled.

[Specialized writer]
(setf (chunked-stream-output-chunking-p (stream chunked-output-stream)) new-value)

This function is used to switch output chunking on stream on or off.

[Specialized reader]
chunked-input-stream-extensions (stream chunked-input-stream) => extensions

Returns an alist of attribute/value pairs corresponding to the optional "chunk extensions" which might have been encountered when reading from stream.

[Specialized reader]
chunked-input-stream-trailers (stream chunked-input-stream) => trailers

Returns the optional "trailer" HTTP headers which might have been sent after the last chunk, i.e. directly before input chunking ended on stream. The format of trailers is identical to that returned by READ-HTTP-HEADERS.

Conditions

Here are two conditions which might be signaled if something bad happens while reading from a chunked stream:

[Condition type]
input-chunking-body-corrupted

A condition of this type is signaled if an unexpected character (octet) is read while reading from a chunked stream with input chunking enabled. This is a subtype of STREAM-ERROR, so STREAM-ERROR-STREAM can be used to access the offending stream.

[Condition type]
input-chunking-unexpected-end-of-file

A condition of this type is signaled if we reach an unexpected EOF on a chunked stream with input chunking enabled. This is a subtype of STREAM-ERROR, so STREAM-ERROR-STREAM can be used to access the offending stream.

RFC 2616 parsing

Chunga needs to know a bit about RFC 2616 syntax in order to cope with extensions and trailers. As these functions are in there anyway, they're exported, so they can be used by other code like for example Drakma.

[Function]
read-line* stream &optional log-stream => line

Reads and assembles characters from the stream stream until a carriage return is read. Makes sure that the following character is a linefeed. Returns the string of characters read excluding the line break. Additonally logs this string to log-stream if it is not NIL.

[Function]
read-http-headers stream &optional log-stream => headers

Reads HTTP header lines from the stream stream (except for the initial status line which is supposed to be read already) and returns a corresponding alist of names and values where the names are keywords and the values are strings. Multiple lines with the same name are combined into one value, the individual values separated by commas. Header lines which are spread across multiple lines are recognized and treated correctly. Additonally logs the header lines to log-stream if it is not NIL.

[Function]
read-token stream => token

Read characters from the stream stream while they are token constituents (according to RFC 2616). It is assumed that there's a token character at the current position. The token read is returned as a string. Doesn't signal an error (but simply stops reading) if END-OF-FILE is encountered after the first character.

[Function]
read-name-value-pair stream &key value-required-p cookie-syntax => pair

Reads a typical (in RFC 2616) name/value or attribute/value combination from the stream stream - a token followed by a #\= character and another token or a quoted string. Returns a cons of the name and the value, both as strings. If value-required-p is NIL (the default is T), the #\= sign and the value are optional. If cookie-syntax is true (the default is NIL), the value is read like the value of a cookie header.

[Function]
read-name-value-pairs stream &key value-required-p cookie-syntax => pairs

Uses READ-NAME-VALUE-PAIR to read and return an alist of name/value pairs from the stream stream. It is assumed that the pairs are separated by semicolons and that the first char read (except for whitespace) will be a semicolon. The parameters are used as in READ-NAME-VALUE-PAIR. Stops reading in case of END-OF-FILE (instead of signaling an error).

[Function]
assert-char stream expected-char => char

Reads the next character from the stream stream and checks if it is the character expected-char. Signals an error otherwise.

[Function]
skip-whitespace stream => char-or-nil

Consume characters from the stream stream until an END-OF-FILE is encountered or a non-whitespace (according to RFC 2616) characters is seen. This character is returned (or NIL in case of END-OF-FILE).

[Function]
trim-whitespace string => string'

Returns a version of the string string where spaces and tab characters are trimmed from the start and the end.

[Special variable]
*current-error-message*

Used by the parsing functions in this section as an introduction to a standardized error message. Must be a string if one of these functions is called.

Acknowledgements

Thanks to Jochen Schmidt's chunking code in ACL-COMPAT for inspiration. This documentation was prepared with DOCUMENTATION-TEMPLATE.

$Header: /usr/local/cvsrep/chunga/doc/index.html,v 1.12 2006/10/26 01:26:38 edi Exp $

BACK TO MY HOMEPAGE