See the first post in The Pragmatic Programmer 20th Anniversary Edition series for an introduction.
Challenge 1
Could some of the requirements of your current project be expressed in a domain-specific language? Would it be possible to write a compiler or translator that could generate most of the code required?
Yes and it is something that my team an I would like to investigate actually implementing. The current project is a REST API and the requirements in question are user authorisation. There are a number of different user types identified by the business - guest, integration, super, administrator, e.t.c. Each of these has a different set of permissions governing which API actions can be performed i.e.:
guest: GET /foo, GET /bar
integration: ANY /foo, POST /bar
super: ANY /foo, ANY /bar
administrator: ANY /foo, ANY /bar, ANY /baz
If there were only the small set of actions and user types above, then a Domain Language would not be worth the effort. However, the API contains a large number of endpoints, user types and permissions/endpoints are (of course) going to change over time. The Domain Language would provide a simple specification for these permissions and generate both the django/django REST framework code and the corresponding tests. Since YAML is used for other tools utilised in the project (OpenAPI, CircleCI e.t.c) the Domain Language could use a YAML for consistency. A very simple example:
Domain Language Specification
/foo:
GET:
- guest
ANY:
- integration
- super
- administrator
/bar:
GET:
- guest
POST:
- integration
ANY:
- super
- administrator
/baz:
ANY:
- administrator
Example /foo DRF ViewSet
Using django-rest-framework-roles to implement role-based permissions.
from rest_framework.viewsets import ModelViewSet
from rest_framework.exceptions import PermissionDenied
from drf_roles.mixins import RoleViewSetMixin
from api.models import Foo
from api.serializers import FooSerializer
class FooViewSet(RoleViewSetMixin, ModelViewSet):
queryset = Foo.objects.all()
serializer_class = FooSerializer
def perform_create_for_guest_user(self, serializer):
raise PermissionDenied()
def perform_update_for_guest_user(self, serializer):
raise PermissionDenied()
def perform_destroy_for_guest_user(self):
raise PermissionDenied()
# can now implement non-boilerplate code if needed
Example /foo Permissions Tests
Using pytest-django.
import pytest
from api.models import User
from django.contrib.auth.models import Group
from rest_framework.exceptions import PermissionDenied
from django.urls import reverse
@pytest.mark.django_db
class TestFoo:
def test_guest_can_perform_GET(self, client):
username = "test"
password = "1234"
user = User.objects.create(username=username, password=password)
user.groups.add(Group.objects.get(name="guest"))
client.login(username=username, password=password)
r = client.get(reverse("foo-list"))
assert r.status_code == 200
with pytest.raises(PermissionDenied):
client.get(reverse("foo-list"))
def test_guest_cannot_perform_POST(self, client):
# ...
# ...
I have skipped over a lot of details here, however the general idea is there and it should be possible to write a compiler for.
Challenge 2
If you decide to adopt mini-languages as a way of programming closer to the problem domain, you’re accepting that some effort will be required to implement them. Can you see ways in which the framework you develop for one project can be reused in others?
The example given above is quite specific, however it could be re-used in another project using django-rest-framework which also requires role-based permissions. Having said that, the Domain Language itself could be used across language/frameworks by implementing different compilers - i.e. for Flask or Phoenix.
Exercise 4
We want to implement a mini-language to control a simple drawing package (perhaps a turtle-graphics system). The language consists of single-letter commands. Some commands are followed by a single number. For example, the following input would draw a rectangle:
P 2 # select pen 2
D # pen down
W 2 # draw west 2cm
N 1 # then north 1
E 2 # then east 2
S 1 # then back south
U # pen up
Implement the code that parses this language. It should be designed so that it is simple to add new commands.
Here’s a simple Python implementation - to focus in on the Domain Language parsing the code to actually perform ‘turtle actions’ is not included. It includes some rudimentary error handling and can be easily extended by adding new entries to the COMMANDS dictionary.
from collections import namedtuple
from typing import Iterable
Command = namedtuple("Command", ["token", "handler", "has_arg"])
# assume handler functions are implemented elsewhere
COMMANDS = {
c.token: c
for c in (
Command("P", select_pen, True),
Command("U", pen_up, False),
Command("D", pen_down, False),
Command("N", move_pen, True),
Command("E", move_pen, True),
Command("S", move_pen, True),
Command("W", move_pen, True),
)
}
def get_arg(line: str) -> int:
""" Retrieve integer argument from input ``line``.
Raises:
ValueError: If no integer argument present in ``line``
"""
try:
return int(line[2])
except (IndexError, ValueError):
raise ValueError(f"Command requires integer argument: {line}")
def parse(lines: Iterable[str]):
for l in lines:
cmd = COMMANDS.get(l[0])
if cmd is None:
raise Exception(f"Unknown command: {l}")
if cmd.has_arg:
cmd.handler(get_arg(l))
else:
cmd.handler()
Exercise 5
In the previous exercise we implemented a simple parser for the turtle graphics language—it was an external domain language. Now implement it again as an internal language. Don’t do anything clever: just write a function for each of the commands. You may have to change the names of the commands to be lower case, and you may need to wrap them inside something to provide some context.
My solution for the previous exercise actually also answers this - the Command.handler functions used to perform actions in the external domain language are the internal language. Here’s the example input from the previous exercise re-written using the internal language:
from enum import Enum, auto
class Direction(Enum):
NORTH = auto()
EAST = auto()
SOUTH = auto()
WEST = auto()
select_pen(2)
pen_down()
move_pen(Direction.WEST, 2)
move_pen(Direction.NORTH, 1)
move_pen(Direction.EAST, 2)
move_pen(Direction.SOUTH, 1)
pen_up()
Exercise 6
Design a BNF grammar to parse a time specification. All of the following examples should be accepted:
4pm, 7:38pm, 23:42, 3:16, 3:16am
<hour-tens-place> and <minute-tens-place> constrain the possible values for hours and minutes to 00 -> 23 and 00 -> 59 respectively:
<time> ::= <hour> <period> | <hour> ":" <minute> <period> | <hour> ":" <minute>
<period> ::= "am" | "pm"
<hour> ::= <hour-tens-place> <digit> | <digit>
<minute> ::= <minute-tens-place> <digit> | <digit>
<hour-tens-place> ::= "0"| "1" | "2"
<minute-tens-place> ::= "0" | "1" | "2" | "3" | "4" | "5"
<digit> ::= "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
Exercise 7
Implement a parser for the BNF grammar in the previous exercise using a PEG parser generator in the language of your choice. The output should be an integer containing the number of minutes past midnight.
Clojure implementation using the instaparse library:
(ns tpp-14-ex7.core
(:require [instaparse.core :as insta]))
(def time-spec
(insta/parser
"time = hour period | hour ':' minute period | hour ':' minute
period = \"am\" | \"pm\"
hour = hour-tens-place digit | digit
minute = minute-tens-place digit | digit
hour-tens-place = '0' | '1' | '2'
minute-tens-place = '0' | '1' | '2' | '3' | '4' | '5'
digit = '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9'"))
(time-spec "4pm")
; => [:time [:hour [:digit "4"]] [:period "pm"]]
(time-spec "7:38pm")
; => [:time [:hour [:digit "7"]] ":" [:minute [:minute-tens-place "3"] [:digit "8"]] [:period "pm"]]
(time-spec "23:42")
; => [:time [:hour [:hour-tens-place "2"] [:digit "3"]] ":" [:minute [:minute-tens-place "4"] [:digit "2"]]]
(time-spec "3:16")
; => [:time [:hour [:digit "3"]] ":" [:minute [:minute-tens-place "1"] [:digit "6"]]]
(time-spec "3:16am")
; => [:time [:hour [:digit "3"]] ":" [:minute [:minute-tens-place "1"] [:digit "6"]] [:period "am"]]
Exercise 8
Good old regex! Here’s a Python implementation that converts a time string into the number of minutes since midnight, along with some basic tests:
import re
import pytest
period = r"(?P<period>am|pm)"
hour = r"(?P<hour>([0-2]\d)|\d)"
minute = r"(?P<minute>([0-5]\d)|\d)"
TIME_SPECS = [
re.compile(p)
for p in [
fr"\A{hour}{period}\Z",
fr"\A{hour}:{minute}{period}\Z",
fr"\A{hour}:{minute}\Z",
]
]
def time_to_minutes(s: str) -> int:
for spec in TIME_SPECS:
m = spec.match(s)
if m:
parts = m.groupdict()
return sum(
(
int(parts["hour"]) * 60,
int(parts.get("minute", 0)),
12 * 60 if parts.get("period") == "pm" else 0,
)
)
@pytest.mark.parametrize(
"time,expected",
[
("4pm", 960),
("7:38pm", 1178),
("23:42", 1422),
("3:16", 196),
("3:16am", 196),
("3", None),
("3:am", None),
("31:30", None),
("10:65", None),
],
)
def test_time_to_minutes(time, expected):
assert time_to_minutes(time) == expected
Note, this does match the BNF grammar, however there are some missing constraints. Both the grammar and parsing code will accept time strings with an hour greater than 23 - ‘overflowing’ the end of the day.
>>> time_to_minutes("24:30")
1470