module documentation

This module implements AMP, the Asynchronous Messaging Protocol.

AMP is a protocol for sending multiple asynchronous request/response pairs over the same connection. Requests and responses are both collections of key/value pairs.

AMP is a very simple protocol which is not an application. This module is a "protocol construction kit" of sorts; it attempts to be the simplest wire-level implementation of Deferreds. AMP provides the following base-level features:

  • Asynchronous request/response handling (hence the name)
  • Requests and responses are both key/value pairs
  • Binary transfer of all data: all data is length-prefixed. Your application will never need to worry about quoting.
  • Command dispatching (like HTTP Verbs): the protocol is extensible, and multiple AMP sub-protocols can be grouped together easily.

The protocol implementation also provides a few additional features which are not part of the core wire protocol, but are nevertheless very useful:

  • Tight TLS integration, with an included StartTLS command.
  • Handshaking to other protocols: because AMP has well-defined message boundaries and maintains all incoming and outgoing requests for you, you can start a connection over AMP and then switch to another protocol. This makes it ideal for firewall-traversal applications where you may have only one forwarded port but multiple applications that want to use it.

Using AMP with Twisted is simple. Each message is a command, with a response. You begin by defining a command type. Commands specify their input and output in terms of the types that they expect to see in the request and response key-value pairs. Here's an example of a command that adds two integers, 'a' and 'b':

    class Sum(amp.Command):
        arguments = [('a', amp.Integer()),
                     ('b', amp.Integer())]
        response = [('total', amp.Integer())]

Once you have specified a command, you need to make it part of a protocol, and define a responder for it. Here's a 'JustSum' protocol that includes a responder for our 'Sum' command:

    class JustSum(amp.AMP):
        def sum(self, a, b):
            total = a + b
            print 'Did a sum: %d + %d = %d' % (a, b, total)
            return {'total': total}
        Sum.responder(sum)

Later, when you want to actually do a sum, the following expression will return a Deferred which will fire with the result:

    ClientCreator(reactor, amp.AMP).connectTCP(...).addCallback(
        lambda p: p.callRemote(Sum, a=13, b=81)).addCallback(
            lambda result: result['total'])

Command responders may also return Deferreds, causing the response to be sent only once the Deferred fires:

    class DelayedSum(amp.AMP):
        def slowSum(self, a, b):
            total = a + b
            result = defer.Deferred()
            reactor.callLater(3, result.callback, {'total': total})
            return result
        Sum.responder(slowSum)

This is transparent to the caller.

You can also define the propagation of specific errors in AMP. For example, for the slightly more complicated case of division, we might have to deal with division by zero:

    class Divide(amp.Command):
        arguments = [('numerator', amp.Integer()),
                     ('denominator', amp.Integer())]
        response = [('result', amp.Float())]
        errors = {ZeroDivisionError: 'ZERO_DIVISION'}

The 'errors' mapping here tells AMP that if a responder to Divide emits a ZeroDivisionError, then the other side should be informed that an error of the type 'ZERO_DIVISION' has occurred. Writing a responder which takes advantage of this is very simple - just raise your exception normally:

    class JustDivide(amp.AMP):
        def divide(self, numerator, denominator):
            result = numerator / denominator
            print 'Divided: %d / %d = %d' % (numerator, denominator, total)
            return {'result': result}
        Divide.responder(divide)

On the client side, the errors mapping will be used to determine what the 'ZERO_DIVISION' error means, and translated into an asynchronous exception, which can be handled normally as any Deferred would be:

    def trapZero(result):
        result.trap(ZeroDivisionError)
        print "Divided by zero: returning INF"
        return 1e1000
    ClientCreator(reactor, amp.AMP).connectTCP(...).addCallback(
        lambda p: p.callRemote(Divide, numerator=1234,
                               denominator=0)
        ).addErrback(trapZero)

For a complete, runnable example of both of these commands, see the files in the Twisted repository:

    doc/core/examples/ampserver.py
    doc/core/examples/ampclient.py

On the wire, AMP is a protocol which uses 2-byte lengths to prefix keys and values, and empty keys to separate messages:

    <2-byte length><key><2-byte length><value>
    <2-byte length><key><2-byte length><value>
    ...
    <2-byte length><key><2-byte length><value>
    <NUL><NUL>                  # Empty Key == End of Message

And so on. Because it's tedious to refer to lengths and NULs constantly, the documentation will refer to packets as if they were newline delimited, like so:

    C: _command: sum
    C: _ask: ef639e5c892ccb54
    C: a: 13
    C: b: 81

    S: _answer: ef639e5c892ccb54
    S: total: 94

Notes:

In general, the order of keys is arbitrary. Specific uses of AMP may impose an ordering requirement, but unless this is specified explicitly, any ordering may be generated and any ordering must be accepted. This applies to the command-related keys _command and _ask as well as any other keys.

Values are limited to the maximum encodable size in a 16-bit length, 65535 bytes.

Keys are limited to the maximum encodable size in a 8-bit length, 255 bytes. Note that we still use 2-byte lengths to encode keys. This small redundancy has several features:

  • If an implementation becomes confused and starts emitting corrupt data, or gets keys confused with values, many common errors will be signalled immediately instead of delivering obviously corrupt packets.
  • A single NUL will separate every key, and a double NUL separates messages. This provides some redundancy when debugging traffic dumps.
  • NULs will be present at regular intervals along the protocol, providing some padding for otherwise braindead C implementations of the protocol, so that <stdio.h> string functions will see the NUL and stop.
  • This makes it possible to run an AMP server on a port also used by a plain-text protocol, and easily distinguish between non-AMP clients (like web browsers) which issue non-NUL as the first byte, and AMP clients, which always issue NUL as the first byte.
Interface IArgumentType An IArgumentType can serialize a Python object into an AMP box and deserialize information from an AMP box back into a Python object.
Interface IBoxReceiver An application object which can receive AmpBox objects and dispatch them appropriately.
Interface IBoxSender A transport which can send AmpBox objects.
Interface IResponderLocator An application object which can look up appropriate responder methods for AMP commands.
Class AMP This protocol is an AMP connection. See the module docstring for protocol details.
Class AmpBox I am a packet in the AMP protocol, much like a regular bytes:bytes dictionary.
Class AmpList Convert a list of dictionaries into a list of AMP boxes on the wire.
Class Argument Base-class of all objects that take values from Amp packets and convert them into objects for Python functions.
Class BinaryBoxProtocol A protocol for receiving AmpBoxes - key/value pairs - via length-prefixed strings. A box is composed of:
Class Boolean Encode True or False as "True" or "False" on the wire.
Class BoxDispatcher A BoxDispatcher dispatches '_ask', '_answer', and '_error' AmpBoxes, both incoming and outgoing, to their appropriate destinations.
Class Command Subclass me to specify an AMP Command.
Class CommandLocator A CommandLocator is a collection of responders to AMP Commands, with the help of the Command.responder decorator.
Class DateTime Encodes datetime.datetime instances.
Class Decimal Encodes decimal.Decimal instances.
Class Descriptor Encode and decode file descriptors for exchange over a UNIX domain socket.
Class Float Encode floating-point values on the wire as their repr.
Class Integer Encode any integer values of any size on the wire as the string representation.
Class ListOf Encode and decode lists of instances of a single other argument type.
Class Path Encode and decode filepath.FilePath instances as paths on the wire.
Class ProtocolSwitchCommand Use this command to switch from something Amp-derived to a different protocol mid-connection. This can be useful to use amp as the connection-startup negotiation phase. Since TLS is a different layer entirely, you can use Amp to negotiate the security parameters of your connection, then switch to a different protocol, and the connection will remain secured.
Class QuitBox I am an AmpBox that, upon being sent, terminates the connection.
Class SimpleStringLocator Implement the AMP.locateResponder method to do simple, string-based dispatch.
Class StartTLS Use, or subclass, me to implement a command that starts TLS.
Class String Don't do any conversion at all; just pass through 'str'.
Class Unicode Encode a unicode string on the wire as UTF-8.
Exception AmpError Base class of all Amp-related exceptions.
Exception BadLocalReturn A bad value was returned from a local command; we were unable to coerce it.
Exception IncompatibleVersions It was impossible to negotiate a compatible version of the protocol with the other end of the connection.
Exception InvalidSignature You didn't pass all the required arguments.
Exception MalformedAmpBox This error indicates that the wire-level protocol was malformed.
Exception NoEmptyBoxes You can't have empty boxes on the connection. This is raised when you receive or attempt to send one.
Exception OnlyOneTLS This is an implementation limitation; TLS may only be started once per connection.
Exception ProtocolSwitched Connections which have been switched to other protocols can no longer accept traffic at the AMP level. This is raised when you try to send it.
Exception RemoteAmpError This error indicates that something went wrong on the remote end of the connection, and the error was serialized and transmitted to you.
Exception TooLong One of the protocol's length limitations was violated.
Exception UnhandledCommand A command received via amp could not be dispatched.
Exception UnknownRemoteError This means that an error whose type we can't identify was raised from the other side.
Constant ANSWER Marker for an Answer packet.
Constant ASK Marker for an Ask packet.
Constant COMMAND Marker for a Command packet.
Constant ERROR Marker for an AMP box of error type.
Constant ERROR_CODE Marker for an AMP box containing the code of an error.
Constant ERROR_DESCRIPTION Marker for an AMP box containing the description of the error.
Constant MAX_KEY_LENGTH Undocumented
Constant MAX_VALUE_LENGTH The maximum length of a message.
Constant PROTOCOL_ERRORS Undocumented
Constant PYTHON_KEYWORDS Undocumented
Constant UNHANDLED_ERROR_CODE Undocumented
Constant UNKNOWN_ERROR_CODE Undocumented
Class _CommandLocatorMeta This metaclass keeps track of all of the Command.responder-decorated methods defined since the last CommandLocator subclass was defined. It assumes (usually correctly, but unfortunately not necessarily so) that those commands responders were all declared as methods of the class being defined...
Class _CommandMeta Metaclass hack to establish reverse-mappings for 'errors' and 'fatalErrors' as class vars.
Class _DescriptorExchanger _DescriptorExchanger is a mixin for BinaryBoxProtocol which adds support for receiving file descriptors, a feature offered by IUNIXTransport.
Class _LocalArgument Local arguments are never actually relayed across the wire. This is just a shim so that StartTLS can pretend to have some arguments: if arguments acquire documentation properties, replace this with something nicer later.
Class _NoCertificate This is for peers which don't want to use a local certificate. Used by AMP because AMP's internal language is all about certificates and this duck-types in the appropriate place; this API isn't really stable though, so it's not exposed anywhere public.
Class _ParserHelper A box receiver which records all boxes received.
Class _SwitchBox Implementation detail of ProtocolSwitchCommand: I am an AmpBox which sets up state for the protocol to switch.
Class _TLSBox I am an AmpBox that, upon being sent, initiates a TLS connection.
Function _objectsToStrings Convert a dictionary of python objects to an AmpBox, converting through a given arglist.
Function _stringsToObjects Convert an AmpBox to a dictionary of python objects, converting through a given arglist.
Function _wireNameToPythonIdentifier (Private) Normalize an argument name from the wire for use with Python code. If the return value is going to be a python keyword it will be capitalized. If it contains any dashes they will be replaced with underscores.
Type Variable _Self Undocumented
Type Variable _T_Callable Undocumented
Variable _log Undocumented

Marker for an Answer packet.

Value
b'_answer'

Marker for an Ask packet.

Value
b'_ask'

Marker for a Command packet.

Value
b'_command'

Marker for an AMP box of error type.

Value
b'_error'
ERROR_CODE: bytes = (source)

Marker for an AMP box containing the code of an error.

Value
b'_error_code'
ERROR_DESCRIPTION: bytes = (source)

Marker for an AMP box containing the description of the error.

Value
b'_error_description'
MAX_KEY_LENGTH: int = (source)

Undocumented

Value
255
MAX_VALUE_LENGTH: int = (source)

The maximum length of a message.

Value
65535
PROTOCOL_ERRORS = (source)

Undocumented

Value
{UNHANDLED_ERROR_CODE: UnhandledCommand}
PYTHON_KEYWORDS: list[str] = (source)

Undocumented

Value
['and',
 'del',
 'for',
 'is',
 'raise',
 'assert',
 'elif',
...
UNHANDLED_ERROR_CODE: bytes = (source)

Undocumented

Value
b'UNHANDLED'
UNKNOWN_ERROR_CODE: bytes = (source)

Undocumented

Value
b'UNKNOWN'
def _objectsToStrings(objects, arglist, strings, proto): (source)

Convert a dictionary of python objects to an AmpBox, converting through a given arglist.

Parameters
objectsa dict mapping names to python objects
arglista list of 2-tuples of strings and Argument objects, as described in Command.arguments.
strings[OUT PARAMETER] An object providing the dict interface which will be populated with serialized data.
protoan AMP instance.
Returns
The converted dictionary mapping names to encoded argument strings (identical to strings).
def _stringsToObjects(strings, arglist, proto): (source)

Convert an AmpBox to a dictionary of python objects, converting through a given arglist.

Parameters
stringsan AmpBox (or dict of strings)
arglista list of 2-tuples of strings and Argument objects, as described in Command.arguments.
protoan AMP instance.
Returns
the converted dictionary mapping names to argument objects.
def _wireNameToPythonIdentifier(key): (source)

(Private) Normalize an argument name from the wire for use with Python code. If the return value is going to be a python keyword it will be capitalized. If it contains any dashes they will be replaced with underscores.

The rationale behind this method is that AMP should be an inherently multi-language protocol, so message keys may contain all manner of bizarre bytes. This is not a complete solution; there are still forms of arguments that this implementation will be unable to parse. However, Python identifiers share a huge raft of properties with identifiers from many other languages, so this is a 'good enough' effort for now. We deal explicitly with dashes because that is the most likely departure: Lisps commonly use dashes to separate method names, so protocols initially implemented in a lisp amp dialect may use dashes in argument or command names.

Parameters
key:bytesa bytes, looking something like 'foo-bar-baz' or 'from'
Returns
a native string which is a valid python identifier, looking something like 'foo_bar_baz' or 'From'.

Undocumented

Value
TypeVar('_Self')
_T_Callable = (source)

Undocumented

Value
TypeVar('_T_Callable',
        bound=Callable[..., object])

Undocumented