Adding controls and bot logic - Part 3 of Solving Puzlogic puzzle game with Python and OpenCV
In the the second part of the series we added the Vision component for the Puzlogic puzzle solving bot.
In this post we will add a mouse controller as well as handle basic bot logic to get the bot actually solve some of the beginner puzzles.
For controls, as before in the Burrito Bison bot, here we only need the mouse. So I’ll reuse the component, which uses
pynput` for mouse controls.
For the bot logic, we’ll need to use the
Controls components, making use of all 3 to solve a given puzzle.
In this case the bot will rely on the human to set up the game level, handle game UIs, while bot itself will be responsible only for finding the right solution and moving game pieces to the target cells.
As mentioned before, we’ll use
Pynput for mouse controls. We start with the basic outline:
import time from pynput.mouse import Button, Controller as MouseController class Controller: def __init__(self): self.mouse = MouseController() def move_mouse(self, x, y): # ... def left_mouse_drag(self, from, to): # ...
move_mouse(x, y), it needs to move the mouse to the given coordinate on screen.
Pynput would be able to move the mouse in a single frame just by changing
mouse.position attribute, but for me that caused problems in the past, as the game would simply not keep up and may handle such mouse movements unpredictably (e.g. not registering mouse movements at all, or moving it only partially).
And in a way that makes sense. Human mouse movements normally take several hundred milliseconds, not under 1ms.
To mimic such gestures I’ve added a way to smooth out the mouse movement over some time period, e.g.
100ms, by taking a step every so often (e.g. every
def move_mouse(self, x, y): def set_mouse_position(x, y): self.mouse.position = (int(x), int(y)) def smooth_move_mouse(from_x, from_y, to_x, to_y, speed=0.1): steps = 40 sleep_per_step = speed // steps x_delta = (to_x - from_x) / steps y_delta = (to_y - from_y) / steps for step in range(steps): new_x = x_delta * (step + 1) + from_x new_y = y_delta * (step + 1) + from_y set_mouse_position(new_x, new_y) time.sleep(sleep_per_step) return smooth_move_mouse( self.mouse.position, self.mouse.position, x, y )
The second necessary operation is mouse dragging, which means pushing the left mouse button down, moving the mouse to the right position and releasing the left mouse button.
Again, all those steps can be programatically performed in under
1ms, but not all games can keep up, so we’ll spread out the gesture actions over roughly a second.
def left_mouse_drag(self, start, end): delay = 0.2 self.move_mouse(*start) time.sleep(delay) self.mouse.press(Button.left) time.sleep(delay) self.move_mouse(*end) time.sleep(delay) self.mouse.release(Button.left) time.sleep(delay)
And that should be sufficient for the bot to reasonably interact with the game.
The bot component will need to communicate between the other 3 component. For bot logic there are 2 necessary steps:
Solvercomponent needs to get information from
Visionabout available cells, their contents, available pieces in order for
Solverto provide a solution.
- Take the solution from
Solver, and map its results into real-life actions, by moving the mouse via
Controlsto the right positions.
We start by defining the structure:
class Bot: """ Needs to map vision coordinates to solver coordinates """ def __init__(self, vision, controls, solver): self.vision = vision self.controls = controls self.solver = solver
Then we need to a way feed information to
Solver. But we can’t just feed vision information straight into the solver (at least not in the current setup), as it both work with slightly differently structured data.
So we first will need to map vision information into structures that
Solver can understand.
def get_moves(self): return self.solver.solve(self.get_board(), self.get_pieces(), self.get_constraints())
self.vision.get_cells() returns cell information in a list of cells of format
Cell(x, y, width, height, content), but
Solver expects a list of cells in format of
(row, column, piece):
def get_board(self): """ Prepares vision cells for solver """ cells = self.vision.get_cells() return list(map(lambda c: (c.x, c.y, c.content), cells))
self.get_pieces() expects a list of integers representing available pieces. However, vision information returns
Cell(x, y, width, height, content). So we need to map that as well:
def get_pieces(self): """ Prepares vision pieces for solver """ return list(map(lambda p: p.content, self.vision.get_pieces()))
self.get_constraints() currently is not implemented in
Vision component, and instead returns an empty list
. We can just pass that along to the
Solver unchanged for now, but will likely have to change it once constraints are implemented.
def get_constraints(self): """ Prepares vision constraints for solver """ return self.vision.get_constraints()
Now that we have the solution in the form
[(target_row, target_column, target_piece_value), ...], we need to map that back into something that could work for the graphical representation.
We already treat
y of each cell as the “row” and “column”, which works because all cells are arranged in a grid anyway.
Now we only need to find which available pieces to take from which cell, and move those to respective target cells.
def do_moves(self): moves = self.get_moves() board = self.vision.get_game_board() available_pieces = self.vision.get_pieces() def get_available_piece(piece, pieces): target = list(filter(lambda p: p.content == piece, pieces)) remaining_pieces = list(filter(lambda p: p != target, pieces)) return (target, remaining_pieces) for (to_x, to_y, required_piece) in moves: (piece, available_pieces) = get_available_piece(required_piece, available_pieces) # Offset of the game screen within a window + offset of the cell + center of the cell move_from = (board.x + piece.x + piece.w/2, board.y + piece.y + piece.h/2) move_to = (board.x + to_x + piece.w/2, board.y + to_y + piece.h/2) print('Moving', move_from, move_to) self.controls.left_mouse_drag( move_from, move_to )
get_available_piece takes the first one from the vision’s set of pieces, and uses that as a source for the solution.
As for handling coordinates, one thing to not forget is that coordinates point to the top left corner of the corner, and are not always absolute in relation to the OS screen.
E.g. a piece’s
x coordinate is relative to the game screen’s
x coordinate, so to find its absolute coordinate we need to sum the two:
absolute_x = board.x + piece.x.
Top left corner of the piece is also not always usable - the game may not respond. However, if we target the center of the cell - that works reliably.
So we offset the absolute coordinate with half of the cell’s width or height:
absolute_x = board.x + piece.x + piece.width.
Finally running the bot!
Now we have all the tools for the bot to solve basic puzzles of Puzlogic. The last remaining piece is to build a
run.py script putting it all together:
from puzbot.vision import ScreenshotSource, Vision from puzbot.bot import Bot from puzbot.solver import Solver from puzbot.controls import Controller source = ScreenshotSource() vision = Vision(source) solver = Solver() controller = Controller() bot = Bot(vision, controller, solver) print('Checking out the game board') bot.refresh() print('Game board:', vision.get_cells()) print('Game pieces:', vision.get_pieces()) print('Calculating the solution') print('Suggested solution:', bot.get_moves()) print('Performing the solution') bot.do_moves()
Now we can run the bot with
python run.py, making sure that the game window is also visible on the screen. Here’s a demo video:
You can see that the bot needs human help to get through the interfaces, and it can only solve puzzles by itself. Even then, it is able to resolve the first 4 basic puzzles, and just by chance is able to solve the 5th one, as it contains sum constraints - right now the bot cannot handle sum constraints in vision. So it does fail on the 6th puzzle.
Full code mentioned in this post can be found on Github: flakas/puzlogic-bot - bot.py, flakas/puzlogic-bot - controls.py.
If you are interested in the full project, you can also find it on Github: flakas/puzlogic-bot.
Subscribe via RSS