Twisted

Why should you care?

• Super easy to implement apps on top of existing protocols
• async IO, cooperative multitasking without thread-like API
• c10k problem - IO bound applications
• elegant design

History

• Starts at year 2000
• @glyph, the author, working on text-based multiplayer game - Twisted Reality
• originaly in java. Threads deadlocking, many bugs, poor scaling
• Alternative found in python's select module
• Worked much better and gradually evolved into the netwoking engine.

Protocol implementations

• Big advantage is many existing protocols implementations
• like HTTP, IRC, XMPP, SMTP, etc
• using them is just a matter of implementating the prepared interfaces:
  1. derive protocol class
  2. implement event methods

Foreign Loop support

• Integrates with foreign event loops, such as GTK+, Qt, Cocoa
• Allows using Twisted as the networking support layer in GUIs

Main parts

• Reactor (Event) Loop
• Protocols
• Deferred
• Transports
• Factories

flow / object lifetime - TCP Server

1. Transport (with a Factory) listens on TCP socket
2. When client connects, factory creates the Protocol object
3. Transport delivers data into the Protocol
4. Protocol handles the data
5. When connection is terminated, protocol object is destroyed

Transport & protocol separation

• Layer abstraction/encapsulation similar to TCP/IP model
• example: netstrings protocol over TCP transport
• json-rpc protocol over netstrings
• or json-rpc over http transport

Reactor Loop

Reactor Loop

class NaiveReactor:
    def run(self):
        while self._run:
          for t in self.transports:
            t.read()
            sleep(0.001)

class NaiveTransport:
  def read(self):
    try:
      xs = read(self.fd, 512):
    except IOError:
        if e.errno != 11:
            self.protocol.connectionLost()
    else:
      self.protocol.dataReceived(xs)

IRC protocol example

class IRCClient(basic.LineReceiver):
    def userJoined(self, user, channel):
        """Called when I see another user joining a channel.
        """
        pass

    def topicUpdated(self, user, channel, newTopic):
        """In channel, user changed the topic to newTopic.

        Also called when first joining a channel.
        """
        pass

IRC protocol example

class YourIRCBot(IRCClient):
    def userJoined(self, user, channel):
      self.transport.write("Hello " + user)

class BotFactory(t.i.p.ReconnectingClientFactory):
  protol = YourIRCBot

reactor.connectTCP("irc.freenode.net", 6667, BotFactory())
reactor.run()
  

Testability

class Protocol:
  def dataReceived(self, data):
    pass

  def makeConnection(self, transport):
    self.transport = transport

def test_irc_hello():
    p = YourIRCBot()
    p.makeConnection(StringIO())
    p.lineReceived(":bob JOIN #foo")
    ok_(p.transport.getvalue(), "PRIVMSG #foo Hello bob")

Generating events - Deferred

def errback(reason):
    print "Oh noes: {0}".format(reason)
    sys.exit(1)

d = twisted.web.client.Agent().request('GET', url, Headers())
# Deferred returned instad of result
d.addCallback(lambda response: r.deliverBody(YourProtocol()))
# callback to execute once result arrives
d.addErrback(errback)
# or errback in case of failure
d.addBoth(lambda _: reactor.stop())
# and another callback regardless of success / fail

reactor.run()

Deferred callback chains

Caveats

Database access

• Less advanced as most libraries don't provide async API
• SQLAlchemy
  • deferToThread
  • recently (Oct 2013) appeared Alchimia - adds async API to sqlalchemy, but limited
• evennia MUD project somehow using Django ORM
• twisted.enterprise.adbapi asynchronous DB-API v2.0 (raw SQL)
• Twistar Active Record for Twisted

Caveats

Unit Testing - Trial

Caveats

Twisted as a platform

• Most libraries provide only sync API
• Therefore libraries you are used to, may not be usable (or need workarounds)
• Relatively small community and slow progress on new standards

Footnotes

• Twisted at AOSABook aosabook.org/en/twisted.html
• Exhaustive Twisted tutorial at krondo.com http://krondo.com/?page_id=1327

• Slides hosted at yaccz.github.com/slides-twisted-intro/
• @yaccz