In the the second part of the series we added the Vision component for the Puzlogic puzzle solving bot.

In this post we will add a mouse controller as well as handle basic bot logic to get the bot actually solve some of the beginner puzzles.

Planning

For controls, as before in the Burrito Bison bot, here we only need the mouse. So I’ll reuse the component, which uses pynput` for mouse controls.

For the bot logic, we’ll need to use the Solver, Vision and Controls components, making use of all 3 to solve a given puzzle. In this case the bot will rely on the human to set up the game level, handle game UIs, while bot itself will be responsible only for finding the right solution and moving game pieces to the target cells.

Implementation

Mouse Controller

As mentioned before, we’ll use Pynput for mouse controls. We start with the basic outline:

import time
from pynput.mouse import Button, Controller as MouseController

class Controller:
    def __init__(self):
        self.mouse = MouseController()

    def move_mouse(self, x, y):
        # ...

    def left_mouse_drag(self, from, to):
        # ...

For move_mouse(x, y), it needs to move the mouse to the given coordinate on screen.

Pynput would be able to move the mouse in a single frame just by changing mouse.position attribute, but for me that caused problems in the past, as the game would simply not keep up and may handle such mouse movements unpredictably (e.g. not registering mouse movements at all, or moving it only partially). And in a way that makes sense. Human mouse movements normally take several hundred milliseconds, not under 1ms.

To mimic such gestures I’ve added a way to smooth out the mouse movement over some time period, e.g. 100ms, by taking a step every so often (e.g. every 2.5ms in 40 steps).

def move_mouse(self, x, y):
    def set_mouse_position(x, y):
        self.mouse.position = (int(x), int(y))
    def smooth_move_mouse(from_x, from_y, to_x, to_y, speed=0.1):
        steps = 40
        sleep_per_step = speed // steps
        x_delta = (to_x - from_x) / steps
        y_delta = (to_y - from_y) / steps
        for step in range(steps):
            new_x = x_delta * (step + 1) + from_x
            new_y = y_delta * (step + 1) + from_y
            set_mouse_position(new_x, new_y)
            time.sleep(sleep_per_step)
    return smooth_move_mouse(
        self.mouse.position[0],
        self.mouse.position[1],
        x,
        y
    )

The second necessary operation is mouse dragging, which means pushing the left mouse button down, moving the mouse to the right position and releasing the left mouse button. Again, all those steps can be programatically performed in under 1ms, but not all games can keep up, so we’ll spread out the gesture actions over roughly a second.

def left_mouse_drag(self, start, end):
    delay = 0.2
    self.move_mouse(*start)
    time.sleep(delay)
    self.mouse.press(Button.left)
    time.sleep(delay)
    self.move_mouse(*end)
    time.sleep(delay)
    self.mouse.release(Button.left)
    time.sleep(delay)

And that should be sufficient for the bot to reasonably interact with the game.

Bot logic

The bot component will need to communicate between the other 3 component. For bot logic there are 2 necessary steps:

  1. Solver component needs to get information from Vision about available cells, their contents, available pieces in order for Solver to provide a solution.
  2. Take the solution from Solver, and map its results into real-life actions, by moving the mouse via Controls to the right positions.

We start by defining the structure:

class Bot:
    """ Needs to map vision coordinates to solver coordinates """

    def __init__(self, vision, controls, solver):
        self.vision = vision
        self.controls = controls
        self.solver = solver

Then we need to a way feed information to Solver. But we can’t just feed vision information straight into the solver (at least not in the current setup), as it both work with slightly differently structured data. So we first will need to map vision information into structures that Solver can understand.

def get_moves(self):
    return self.solver.solve(self.get_board(), self.get_pieces(), self.get_constraints())

self.vision.get_cells() returns cell information in a list of cells of format Cell(x, y, width, height, content), but Solver expects a list of cells in format of (row, column, piece):

def get_board(self):
    """ Prepares vision cells for solver """
    cells = self.vision.get_cells()

    return list(map(lambda c: (c.x, c.y, c.content), cells))

self.get_pieces() expects a list of integers representing available pieces. However, vision information returns Cell(x, y, width, height, content). So we need to map that as well:

def get_pieces(self):
    """ Prepares vision pieces for solver """
    return list(map(lambda p: p.content, self.vision.get_pieces()))

self.get_constraints() currently is not implemented in Vision component, and instead returns an empty list []. We can just pass that along to the Solver unchanged for now, but will likely have to change it once constraints are implemented.

def get_constraints(self):
    """ Prepares vision constraints for solver """
    return self.vision.get_constraints()

Now that we have the solution in the form [(target_row, target_column, target_piece_value), ...], we need to map that back into something that could work for the graphical representation. We already treat x and y of each cell as the “row” and “column”, which works because all cells are arranged in a grid anyway. Now we only need to find which available pieces to take from which cell, and move those to respective target cells.

def do_moves(self):
    moves = self.get_moves()
    board = self.vision.get_game_board()
    available_pieces = self.vision.get_pieces()

    def get_available_piece(piece, pieces):
        target = list(filter(lambda p: p.content == piece, pieces))[0]
        remaining_pieces = list(filter(lambda p: p != target, pieces))
        return (target, remaining_pieces)

    for (to_x, to_y, required_piece) in moves:
        (piece, available_pieces) = get_available_piece(required_piece, available_pieces)

        # Offset of the game screen within a window + offset of the cell + center of the cell
        move_from = (board.x + piece.x + piece.w/2, board.y + piece.y + piece.h/2)
        move_to = (board.x + to_x + piece.w/2, board.y + to_y + piece.h/2)
        print('Moving', move_from, move_to)

        self.controls.left_mouse_drag(
            move_from,
            move_to
        )

get_available_piece takes the first one from the vision’s set of pieces, and uses that as a source for the solution.

As for handling coordinates, one thing to not forget is that coordinates point to the top left corner of the corner, and are not always absolute in relation to the OS screen. E.g. a piece’s x coordinate is relative to the game screen’s x coordinate, so to find its absolute coordinate we need to sum the two: absolute_x = board.x + piece.x.

Top left corner of the piece is also not always usable - the game may not respond. However, if we target the center of the cell - that works reliably. So we offset the absolute coordinate with half of the cell’s width or height: absolute_x = board.x + piece.x + piece.width.

Finally running the bot!

Now we have all the tools for the bot to solve basic puzzles of Puzlogic. The last remaining piece is to build a run.py script putting it all together:

from puzbot.vision import ScreenshotSource, Vision
from puzbot.bot import Bot
from puzbot.solver import Solver
from puzbot.controls import Controller

source = ScreenshotSource()
vision = Vision(source)
solver = Solver()
controller = Controller()
bot = Bot(vision, controller, solver)

print('Checking out the game board')
bot.refresh()

print('Game board:', vision.get_cells())
print('Game pieces:', vision.get_pieces())

print('Calculating the solution')
print('Suggested solution:', bot.get_moves())
print('Performing the solution')
bot.do_moves()

Now we can run the bot with python run.py, making sure that the game window is also visible on the screen. Here’s a demo video:

You can see that the bot needs human help to get through the interfaces, and it can only solve puzzles by itself. Even then, it is able to resolve the first 4 basic puzzles, and just by chance is able to solve the 5th one, as it contains sum constraints - right now the bot cannot handle sum constraints in vision. So it does fail on the 6th puzzle.

Full code mentioned in this post can be found on Github: flakas/puzlogic-bot - bot.py, flakas/puzlogic-bot - controls.py.

If you are interested in the full project, you can also find it on Github: flakas/puzlogic-bot.