Sitemap / Advertise

Introduction

Via OpenCV, recognize villagers to display their birthdays, loves, likes, and hates while playing Stardew Valley on LattePanda Alpha.


Tags

Share

Stardew Valley Villager Recognition and Gift Preferences Bot w/ OpenCV

Advertisement:


read_later

Read Later



read_later

Read Later

Introduction

Via OpenCV, recognize villagers to display their birthdays, loves, likes, and hates while playing Stardew Valley on LattePanda Alpha.

Tags

Share





Advertisement

Advertisement




    Components :
  • [1]LattePanda Alpha 864s
  • [1]Arduino Leonardo (Built-in)
  • [1]DFRobot 8.9" 1920x1200 IPS Touch Display
  • [1]3.5" 320x480 TFT LCD Touch Screen (ILI9488)
  • [2]Logic Level Converter (Bi-Directional)
  • [1]Buzzer
  • [1]5mm Green LED
  • [1]220Ω Resistor
  • [1]Breadboard
  • [1]Jumper Wires

Description

In this project, I wanted to focus on an emerging field for improving our overall experience while playing and exploring video games, known as computer vision implementation in video games. We have nearly interminable options while employing computer vision to improve video games, from aim assistance bots for first-person shooter (FPS) games to mining bots for role-playing (RPG) games. Although computer vision implementation in video games is commonly utilized for competitive online games, I decided to employ computer vision in one of my favorite video games - Stardew Valley.

In Stardew Valley, even though I played it more than once already, I still needed to scrutinize guides to remember villagers' gift preferences while trying to increase my friendship with certain characters. Thus, I decided to employ computer vision to recognize villagers and display their birthdays and gift preferences (loves, likes, and hates) automatically while playing Stardew Valley.

Since I wanted this project to be as compact as possible, I chose to use a LattePanda Alpha 864s, a high-performance SBC (single-board computer), has an embedded Arduino Leonardo. To recognize villagers while exploring the Stardew Valley world, I deployed the OpenCV template matching. After detecting villagers successfully, to display their gift preferences and birthdays without interrupting the gaming experience or performance, I used the embedded Arduino Leonardo on the LattePanda Alpha. Arduino Leonardo prints the detected villager information on a 3.5" 320x480 TFT LCD Touch Screen (ILI9488) and notifies the player via a buzzer and a 5mm green LED.

Because I was usually struggling to increase my friendship with these eight characters below, I used the OpenCV template matching to recognize them:

After running this project and obtaining accurate results, it obviates the need for shrewd gift acumen for villagers while playing Stardew Valley. 😃 🎁

Huge thanks to DFRobot for sponsoring this project.

Sponsored products by DFRobot:

⭐ LattePanda Alpha 864s | Inspect

⭐ DFRobot 8.9" 1920x1200 IPS Touch Display | Inspect

project-image
Figure - 69.1


project-image
Figure - 69.2


project-image
Figure - 69.3


project-image
Figure - 69.4

Step 1: Recognizing villagers while playing Stardew Valley with OpenCV

I decided to employ template matching to recognize villagers in Stardew Valley while playing the game. Template Matching is a method for searching and finding the location of a template image in a larger (input) image. OpenCV provides a built-in function - cv.matchTemplate() - for deploying template matching. It basically slides the template image over the input image (as in 2D convolution) and compares the patches of the input image against the template image to find the overlapped patches. When the function finds the overlapped patches, it saves the comparison results. If the input image is of size (W x H) and the template image is of size (w x h), the output (results) image will have a size of (W - w + 1, H - h + 1). You can get more information regarding template matching from this guide.

There are several comparison methods implemented in OpenCV, and each comparison method applies a unique formula to generate results. You can inspect TemplateMatchModes to get more information regarding comparison methods, describing the formulae for the available comparison methods.

I developed an application in Python to recognize villagers in Stardew Valley and send their information (gift preferences and birthdays) to the Arduino Leonardo. As shown below, the application consists of one folder and five code files:

I will elucidate each file in detail in the following steps.

To execute my application to recognize villagers successfully, I installed the required modules on Thonny. If you want, you can use a different Python IDE.

⭐ Download the opencv-python module.

pip install opencv-python

project-image
Figure - 69.5

⭐ Download the pywin32 module, which provides access to Windows APIs.

pip install pywin32

project-image
Figure - 69.6

Step 1.1: Deriving templates from in-game screenshots and defining villager preferences

First of all, to be able to use template matching to detect villagers in Stardew Valley, I needed to create templates for each villager (character) in my list shown above. Since there are different poses in Stardew Valley for characters, I utilized in-game screenshots to catch unique character poses - sitting, turning right, turning left, going forward, etc. Because I was running Stardew Valley on Steam on LattePanda Alpha, I took screenshots while playing the game on Steam by pressing F12.

project-image
Figure - 69.7


project-image
Figure - 69.8


project-image
Figure - 69.9

Then, I cropped character poses from the in-game screenshots to derive my villager templates.

project-image
Figure - 69.10


project-image
Figure - 69.11


project-image
Figure - 69.12

After collecting templates (cropped character poses) for each villager in my list, I saved them in the assets folder. Then, I created a dictionary (character_pics) containing template paths for each villager and stored it in the pics.py file.

project-image
Figure - 69.13


project-image
Figure - 69.14

Finally, I created a JSON file (villagers.json) to store gift preferences and birthdays for each villager in my list:


	"Abigail": {
		"birthday" : "Fall 13",
		"loves" : "Amethyst, Banana Pudding, Blackberry Cobbler, Chocolate Cake, Pufferfish, Pumpkin, Spicy Eel",
		"likes" : "Quartz",
		"hates" : "Clay, Holly",
		"introduction" : "Abigail is a villager who lives at Pierre's General Store in Pelican Town. She is one of the twelve characters available to marry. "
	},

...


project-image
Figure - 69.15

Step 1.2: Capturing real-time game window data (frames)

I needed to elicit real-time frames (screenshots) from the game window while playing Stardew Valley to employ template matching to recognize the villagers in my list. Even though there are several methods to capture screenshots in Python, I decided to utilize the Windows (Win32) API because it can grasp the screen data for only the game window faster than other methods. Also, it can capture frames (screenshots) of the game window when it is not full-screen.

I created a class named captureWindowFrame in the windowframe.py file to obtain real-time game window data (frames) while playing Stardew Valley and convert the raw frame data to process it with OpenCV.

⭐ Do not forget to install the pywin32 module to access the Windows (Win32) API.

⭐ In the __init__ function, define the handle for the game window, get the window sizes, and adjust them to remove redundant pixels from captured frames.


    def __init__(self, window_name):
        # Define the handle for the selected window.
        # Full Screen:
        # self.hwnd = win32gui.GetDesktopWindow()
        self.hwnd = win32gui.FindWindow(None, window_name)
        if not self.hwnd:
            raise Exception('Window Not Found: {}'.format(window_name))
        # Window Sizes:
        window_rect = win32gui.GetWindowRect(self.hwnd)
        self.w = window_rect[2] - window_rect[0]
        self.h = window_rect[3] - window_rect[1]
        # Adjust the window to remove unnecessary pixels:
        self.border_px = 8
        self.titlebar_px = 30
        self.w = self.w - (self.border_px * 2)
        self.h = self.h - self.titlebar_px - self.border_px

⭐ In the get_frames function, capture a new frame from the game window with the Win32 API, convert the raw frame data to process it with OpenCV, then clear the raw frame data.

⭐ Before returning the converted image (frame) data, drop the alpha channel and make the image (img) C_CONTIGUOUS to avoid errors while applying template matching.


    def get_frames(self):
        # Get the frame (screenshot) data:
        wDC = win32gui.GetWindowDC(self.hwnd)
        dcObj = win32ui.CreateDCFromHandle(wDC)
        cDC = dcObj.CreateCompatibleDC()
        dataBitMap = win32ui.CreateBitmap()
        dataBitMap.CreateCompatibleBitmap(dcObj, self.w, self.h)
        cDC.SelectObject(dataBitMap)
        cDC.BitBlt((0, 0), (self.w, self.h), dcObj, (self.border_px, self.titlebar_px), win32con.SRCCOPY)
        # Convert the raw frame data for OpenCV:
        signedIntsArray = dataBitMap.GetBitmapBits(True)
        img = np.fromstring(signedIntsArray, dtype='uint8')
        img.shape = (self.h, self.w, 4)
        # Clear the frame data:
        dcObj.DeleteDC()
        cDC.DeleteDC()
        win32gui.ReleaseDC(self.hwnd, wDC)
        win32gui.DeleteObject(dataBitMap.GetHandle())
        # Drop the alpha channel and make the image C_CONTIGUOUS to avoid errors while running OpenCV:
        img = img[...,:3]
        img = np.ascontiguousarray(img)
        # Return:
        return img

⭐ In the get_window_names function, print the list of window names and hex codes if necessary.


    def get_window_names(self):
        def winEnumHandler(hwnd, ctx):
            if win32gui.IsWindowVisible(hwnd):
                print(hex(hwnd), win32gui.GetWindowText(hwnd))
        win32gui.EnumWindows(winEnumHandler, None)

project-image
Figure - 69.16


project-image
Figure - 69.17

Step 1.3: Applying real-time object detection to the captured game window frames

After eliciting real-time game window frames (screenshots) and formatting them appropriately for OpenCV, I applied template matching to the captured frames so as to detect one of the eight villagers (characters) in my list:

I created a class named computerVision in the vision.py file to process the captured real-time game window frames (screenshots) with OpenCV by employing template matching. The class searches for all given poses (templates) of a villager in the captured frames and generates results for each template. Also, it presents the real-time villager detection results by drawing rectangles or markers on the captured frames.

I used the TM_CCOEFF_NORMED comparison method (explained above) to generate results. Since template matching (cv.matchTemplate) generates results in confidence scores for each possible match location in the main (input) image, I define a threshold (0.65) to get the most accurate results for multiple objects (templates). You can examine other comparison methods and different threshold values while applying template matching.

⭐ Do not forget to install the opencv-python module to utilize template matching in OpenCV.

⭐ In the __init__ function, read image data with the cv.imread function from each template (character pose) in the given cropped image list and save template sizes.


    def __init__(self, cropped_images_to_detect):
        # Define parameters:
        self.templates = []
        self.template_sizes = []
        # Get and save the required parameters (template and sizes) for all cropped images in the given list:
        for img in cropped_images_to_detect:
            # Template:
            cropped_img = cv.imread(img, cv.IMREAD_UNCHANGED)
            self.templates.append(cropped_img)
            # Sizes:
            cropped_w = cropped_img.shape[1]
            cropped_h = cropped_img.shape[0]
            self.template_sizes.append((cropped_w, cropped_h))

⭐ In the detect_cropped_image function, employ template matching on the given main image (game window frame) to generate results for each given template (character pose):

⭐ Obtain the detected template (character pose) locations above the given threshold (0.65) on the main image (window frame).

⭐ Zip the detected template locations into (X, Y) position tuples.

⭐ Create the rectangles list [x, y, w, h] from the template locations and sizes. Add each element to the rectangles list twice to avoid excluding single points from the list while grouping.

⭐ Group the rectangles list to avert showing overlapping or close positions by the 0.7 relative difference between sides of the given rectangles.


    def detect_cropped_image(self, main_img, threshold, method=cv.TM_CCOEFF_NORMED):
        self.main_img = main_img
        # Generate results for each template:
        self.rectangles = []
        for i in range(len(self.templates)):
            # Get detected cropped image locations on the main image:
            result = cv.matchTemplate(self.main_img, self.templates[i], method)
            locations = np.where(result >= threshold)
            # Zip the detected cropped image locations into (X, Y) position tuples:
            locations = list(zip(*locations[::-1]))
            # Create the rectangles list [x, y, w, h] to avert overlapping positions:
            for loc in locations:
                w, h = self.template_sizes[i]
                rect = [int(loc[0]), int(loc[1]), int(w), int(h)]
                self.rectangles.append(rect)
                # For single points:
                self.rectangles.append(rect)
        # Group the rectangles list:
        self.rectangles, weights = cv.groupRectangles(self.rectangles, 1, 0.7)

⭐ In the get_and_draw_points function, loop through all detected template data in the rectangles list to define the rectangle box and center points:

⭐ According to the selected result type, draw rectangles (boxes) around the detected template locations or markers on the center points of the detected template locations.

⭐ Then, return the altered main image (game window frame) with boxes or markers and clear the rectangles list to avoid errors. If nothing is detected, return the original frame.

⭐ Modify the built-in parameters in the function to change the color and size settings of boxes and markers.


    def get_and_draw_points(self, result_type, rect_color=(255,0,255), rect_type=cv.LINE_4, rect_thickness=3, marker_color=(255,0,255), marker_type=cv.MARKER_STAR, marker_size=25, marker_thickness=3):
        self.detected_center_points = []
        if(len(self.rectangles)):
            # Loop all detected image locations in the rectangles list:
            for (x, y, w, h) in self.rectangles:
                # Define the rectangle box points:
                top_left = (x, y)
                bottom_right = (x + w, y + h)
                # Define the rectangle center points for drawing markers:
                center_x = x + int(w/2)
                center_y = y + int(h/2)
                # Save the center points:
                self.detected_center_points.append((center_x, center_y))
                # Draw results on the main image:
                if(result_type == 'boxes'):
                    # Draw rectangles around the detected cropped images:
                    cv.rectangle(self.main_img, top_left, bottom_right, color=rect_color, thickness=rect_thickness, lineType=rect_type)
                elif(result_type == 'markers'):
                    # Draw markers on the detected cropped images:
                    cv.drawMarker(self.main_img, (center_x, center_y), marker_color, marker_type, marker_size, marker_thickness)
        return self.main_img
        # Clear the rectangles list to avoid errors:
        self.rectangles = []

⭐ In the get_center_points function, return the center points of the detected template locations.


    def get_center_points(self):
        return self.detected_center_points

project-image
Figure - 69.18


project-image
Figure - 69.19

Step 1.4: Sending the detected villager information to the Arduino Leonardo

As shown above, I created a JSON file (villagers.json) to store gift preferences and birthdays for each villager in my list. After detecting villagers successfully by applying template matching on real-time game window frames, I created a function named arduino_serial_comm in the arduino_serial_comm.py file to send the detected villager information in the villagers.json file to the build-in Arduino Leonardo port via serial communication.

⭐ In the arduino_serial_comm function, define the build-in Arduino Leonardo port (COM6) on the LattePanda Alpha.

⭐ Then, elicit information of the given villager (character) from the villagers.json file and send that information to the Arduino Leonardo via serial communication.

⭐ Do not forget to use the encode function to be able to send strings via serial communication.


def arduino_serial_comm(port, character):
    # Define the data file (json):
    json_file = 'assets/villagers.json'
    # Define the build-in Arduino Leonardo port on the LattePanda:
    arduino = serial.Serial(port, 9600, timeout=1)
    # Elicit information:
    with open (json_file) as data:
        villagers = json.load(data)
        c = villagers[character]
        msg = "Character: {}\n\nBirthday: {}\n\nLoves: {}\n\nLikes: {}\n\nHates: {}".format(character, c['birthday'], c['loves'], c['likes'], c['hates'])
        print(msg)
        arduino.write(msg.encode())
        sleep(1)

project-image
Figure - 69.20

Step 1.5: Initiating and running real-time villager recognition

After creating all the code files mentioned above, I decided to combine all required classes and functions in the main.py file to make my code more coherent.

⭐ Include the required classes and functions.

⭐ Define a villager (character) name among the eight villagers in the given list above.

⭐ Create a list of templates (cropped villager poses) of the given character.

⭐ Define the class objects.

⭐ Define the detection status parameter to avoid sending repetitive messages to the Arduino Leonardo.


import cv2 as cv
import numpy as np
from windowframe import captureWindowFrame
from vision import computerVision
from arduino_serial_comm import arduino_serial_comm
from pics import character_pics

# Define a character (villager) name in Stardew Valley:
_character = 'Penny'
# Create a list of cropped images:
cropped_images_to_detect = [character_pics[_character][0], character_pics[_character][1], character_pics[_character][2]]

# Define the class objects:
winframe = captureWindowFrame('Stardew Valley')
game_vision = computerVision(cropped_images_to_detect)

# Define the detection status to avoid repetitive messages to the Arduino Leonardo:
detection_status = True

⭐ In the detect_character function:

⭐ Get a new game window frame (screenshot).

⭐ Detect template locations in the captured game window frame and generate results.

⭐ Obtain the output image and the center points of the detected template locations.

⭐ If template matching recognizes the villager in the given frame, send the detected villager information to the Arduino Leonardo via serial communication. Unless the detected villager disappears in frames, do not send data again to avoid repetitive messages.

⭐ When recognized, save the output image with the name of the detected villager - cv.imwrite.

⭐ Display the output image - cv.imshow.


def detect_character(character, result_type, port):
    global detection_status
    # Get a new frame (screenshot):
    new_frame = winframe.get_frames()
    # Detect the cropped images in the given list:
    game_vision.detect_cropped_image(new_frame, 0.65)
    # Get the output image:
    output = game_vision.get_and_draw_points(result_type)
    # Get the detected cropped image points:
    points = len(game_vision.get_center_points())
    print(points)
    # Send message (data) to the Arduino Leonardo via serial communication:
    if(points == 0):
        detection_status = True
    elif (points > 0 and detection_status):
        detection_status = False
        arduino_serial_comm(port, character)
        # Save the output image:
        cv.imwrite('result_{}.jpg'.format(character), output)
    # Show the output image:
    cv.imshow('OpenCV Computer Vision on LattePanda', output)

⭐ In the loop, initialize the detect_character function.

⭐ Press 'q' to exit and clear.


while True:
    # Initialize villager (character) recognition:
    detect_character(_character, 'boxes', 'COM6')
    #detect_character(_character, 'markers', 'COM6')
    
    # Press 'q' to exit:
    if(cv.waitKey(1) == ord('q')):
        cv.destroyAllWindows()
        break

project-image
Figure - 69.21


project-image
Figure - 69.22

After running the main.py file successfully:

🎮💻 It opens a new window named OpenCV Computer Vision on LattePanda to show real-time captured game window frames (screenshots) and detection results as a video stream.

project-image
Figure - 69.23

🎮💻 Then, according to the selected result type, it draws rectangles (boxes) around the detected template locations or markers on the center points of the detected template locations.

project-image
Figure - 69.24


project-image
Figure - 69.25


project-image
Figure - 69.26


project-image
Figure - 69.27

🎮💻 Finally, it displays the number of the detected template locations in the given game window frame on the shell and sends the recognized villager information to the Arduino Leonardo via serial communication. As explained above, it sends data for once until the detected villager disappears in frames to avoid repetitive messages.

project-image
Figure - 69.28


project-image
Figure - 69.29

Step 2: Displaying the transferred villager information with the Arduino Leonardo

After completing all steps above and transferring the detected villager information successfully to the embedded Arduino Leonardo on the LattePanda Alpha via serial communication, I utilized a 3.5" 320x480 TFT LCD Touch Screen (ILI9488) to display that information. Also, I used a buzzer and a 5mm green LED to notify the player when template matching recognizes a villager (character).

Download the required libraries to print data on the 320x480 TFT LCD Touch Screen (ILI9488):

ILI9488 | Library

Adafruit_GFX | Library

⭐ Include the required libraries.


#include "SPI.h"
#include <Adafruit_GFX.h>
#include <ILI9488.h>

⭐ Define the required pins for the 320x480 TFT LCD Touch Screen (ILI9488) and initiate it with the hardware SPI on the Arduino Leonardo (SCK, MISO, MOSI).


#define TFT_DC 9
#define TFT_CS 10
#define TFT_RST 8

// Use hardware SPI (on Leonardo, SCK, MISO, MOSI) and the above for DC/CS/RST
ILI9488 tft = ILI9488(TFT_CS, TFT_DC, TFT_RST);

⭐ Initialize serial communication and the 320x480 TFT LCD Touch Screen (ILI9488).


  Serial.begin(9600);
  // Initialize the TFT LCD Touch Screen (ILI9488):
  tft.begin();
  tft.setRotation(1);
  tft.fillScreen(ILI9488_RED);
  tft.setCursor(0, 0);
  tft.setTextColor(ILI9488_BLACK); tft.setTextSize(4);
  tft.println("\nStardew Valley\n");
  tft.println("Villager Recognition\n");
  tft.println("and Gift Preferences\n");
  tft.println("Bot w/ OpenCV\n");

⭐ If there is incoming serial data, save it to the gameData string to obtain the information of the villager detected by template matching.


  while (Serial.available()) {
    char c = (char)Serial.read();
    gameData += c;
  }

⭐ After eliciting the detected villager information, display that information (character name, birthday, loves, likes, hates) on the TFT LCD Touch Screen and notify the player via the buzzer and the 5mm green LED. Then, clear the gameData string.


  if(gameData != ""){
    // Notify the user:
    tft.fillScreen(ILI9488_BLACK);
    tft.setCursor(10, 5);
    tft.setTextColor(ILI9488_RED); tft.setTextSize(4);
    tft.println("Stardew Valley\n");
    tft.setTextColor(ILI9488_GREEN); tft.setTextSize(2);
    tft.println(gameData);
    tone(buzzer, 500);
    digitalWrite(green_LED, HIGH);
    delay(500);
    noTone(buzzer);
    digitalWrite(green_LED, LOW);
  }
  // Clear:
  delay(250);
  gameData = "";

project-image
Figure - 69.30


project-image
Figure - 69.31

After running this code (lattepanda_detection_results_display.ino) on the embedded Arduino Leonardo on the LattePanda Alpha:

🎮💻 It shows the home screen while waiting for incoming serial data.

project-image
Figure - 69.32


project-image
Figure - 69.33

🎮💻 After template matching detects a villager (character) and transfers the detected villager information to the Arduino Leonardo via serial communication, it displays:

project-image
Figure - 69.34


project-image
Figure - 69.35

🎮💻 Also, it notifies the player via the buzzer and the 5mm green LED.

project-image
Figure - 69.36

Results for each villager while exploring Stardew Valley

After completing my project, I explored the Stardew Valley world to detect villagers (characters) in my list with template matching and display their information (birthdays and gift preferences) on the Arduino Leonardo:

As far as my experiments go, template matching runs stupendously on real-time captured game window frames from Stardew Valley:

📌 Abigail

project-image
Figure - 69.37


project-image
Figure - 69.38


project-image
Figure - 69.39

📌 Emily

project-image
Figure - 69.40


project-image
Figure - 69.41


project-image
Figure - 69.42

📌 Haley

project-image
Figure - 69.43


project-image
Figure - 69.44


project-image
Figure - 69.45

📌 Maru

project-image
Figure - 69.46


project-image
Figure - 69.47


project-image
Figure - 69.48

📌 Penny

project-image
Figure - 69.49


project-image
Figure - 69.50


project-image
Figure - 69.51

📌 Alex

project-image
Figure - 69.52


project-image
Figure - 69.53


project-image
Figure - 69.54

📌 Harvey

project-image
Figure - 69.55


project-image
Figure - 69.56


project-image
Figure - 69.57

📌 Sam

project-image
Figure - 69.58


project-image
Figure - 69.59


project-image
Figure - 69.60

Connections and Adjustments


// Connections
// LattePanda Alpha 864s (Arduino Leonardo) :  
//                                3.5'' 320x480 TFT LCD Touch Screen (ILI9488)
// D10 --------------------------- CS 
// D8  --------------------------- RESET 
// D9  --------------------------- D/C
// MOSI -------------------------- SDI 
// SCK --------------------------- SCK 
// MISO -------------------------- SDO(MISO) 
//                                Buzzer
// D11 --------------------------- S     
//                                5mm Green LED
// D12 --------------------------- S

I connected the 3.5" 320x480 TFT LCD Touch Screen (ILI9488), the buzzer, and the 5mm green LED to the embedded Arduino Leonardo via the LattePanda Alpha GPIO pins.

Since the Arduino Leonardo operates at 5V and the TFT LCD Touch Screen (ILI9488) requires 3.3V logic level voltage, I utilized two bi-directional logic level converters to shift the voltage for the connections between the TFT LCD Touch Screen (ILI9488) and the Arduino Leonardo.

project-image
Figure - 69.61

Videos and Conclusion


After completing all steps above, I managed to increase my friendship with Stardew Valley villagers (characters) effortlessly without needing to scrutinize guides so as to remember a certain villager's gift preferences and birthday 😃

project-image
Figure - 69.62

Code

main.py

Download



# Stardew Valley Villager Recognition and Gift Preferences Bot w/ OpenCV
#
# Via OpenCV, recognize villagers to display their birthdays, loves, likes, and hates
# while playing Stardew Valley on LattePanda Alpha.
#
# LattePanda Alpha 864s
#
# By Kutluhan Aktar

import cv2 as cv
import numpy as np
from windowframe import captureWindowFrame
from vision import computerVision
from arduino_serial_comm import arduino_serial_comm
from pics import character_pics

# Define a character (villager) name in Stardew Valley:
_character = 'Penny'
# Create a list of cropped images:
cropped_images_to_detect = [character_pics[_character][0], character_pics[_character][1], character_pics[_character][2]]

# Define the class objects:
winframe = captureWindowFrame('Stardew Valley')
game_vision = computerVision(cropped_images_to_detect)

# Define the detection status to avoid repetitive messages to the Arduino Leonardo:
detection_status = True

def detect_character(character, result_type, port):
    global detection_status
    # Get a new frame (screenshot):
    new_frame = winframe.get_frames()
    # Detect the cropped images in the given list:
    game_vision.detect_cropped_image(new_frame, 0.65)
    # Get the output image:
    output = game_vision.get_and_draw_points(result_type)
    # Get the detected cropped image points:
    points = len(game_vision.get_center_points())
    print(points)
    # Send message (data) to the Arduino Leonardo via serial communication:
    if(points == 0):
        detection_status = True
    elif (points > 0 and detection_status):
        detection_status = False
        arduino_serial_comm(port, character)
        # Save the output image:
        cv.imwrite('result_{}.jpg'.format(character), output)
    # Show the output image:
    cv.imshow('OpenCV Computer Vision on LattePanda', output)

while True:
    # Initialize villager (character) recognition:
    detect_character(_character, 'boxes', 'COM6')
    #detect_character(_character, 'markers', 'COM6')
    
    # Press 'q' to exit:
    if(cv.waitKey(1) == ord('q')):
        cv.destroyAllWindows()
        break
    
print('Window Closed!')


windowframe.py

Download



import numpy as np
import win32gui, win32ui, win32con

class captureWindowFrame:
    def __init__(self, window_name):
        # Define the handle for the selected window.
        # Full Screen:
        # self.hwnd = win32gui.GetDesktopWindow()
        self.hwnd = win32gui.FindWindow(None, window_name)
        if not self.hwnd:
            raise Exception('Window Not Found: {}'.format(window_name))
        # Window Sizes:
        window_rect = win32gui.GetWindowRect(self.hwnd)
        self.w = window_rect[2] - window_rect[0]
        self.h = window_rect[3] - window_rect[1]
        # Adjust the window to remove unnecessary pixels:
        self.border_px = 8
        self.titlebar_px = 30
        self.w = self.w - (self.border_px * 2)
        self.h = self.h - self.titlebar_px - self.border_px
    def get_frames(self):
        # Get the frame (screenshot) data:
        wDC = win32gui.GetWindowDC(self.hwnd)
        dcObj = win32ui.CreateDCFromHandle(wDC)
        cDC = dcObj.CreateCompatibleDC()
        dataBitMap = win32ui.CreateBitmap()
        dataBitMap.CreateCompatibleBitmap(dcObj, self.w, self.h)
        cDC.SelectObject(dataBitMap)
        cDC.BitBlt((0, 0), (self.w, self.h), dcObj, (self.border_px, self.titlebar_px), win32con.SRCCOPY)
        # Convert the raw frame data for OpenCV:
        signedIntsArray = dataBitMap.GetBitmapBits(True)
        img = np.fromstring(signedIntsArray, dtype='uint8')
        img.shape = (self.h, self.w, 4)
        # Clear the frame data:
        dcObj.DeleteDC()
        cDC.DeleteDC()
        win32gui.ReleaseDC(self.hwnd, wDC)
        win32gui.DeleteObject(dataBitMap.GetHandle())
        # Drop the alpha channel and make the image C_CONTIGUOUS to avoid errors while running OpenCV:
        img = img[...,:3]
        img = np.ascontiguousarray(img)
        # Return:
        return img
    # List window names:
    def get_window_names(self):
        def winEnumHandler(hwnd, ctx):
            if win32gui.IsWindowVisible(hwnd):
                print(hex(hwnd), win32gui.GetWindowText(hwnd))
        win32gui.EnumWindows(winEnumHandler, None)
    


vision.py

Download



import cv2 as cv
import numpy as np

class computerVision:
    def __init__(self, cropped_images_to_detect):
        # Define parameters:
        self.templates = []
        self.template_sizes = []
        # Get and save the required parameters (template and sizes) for all cropped images in the given list:
        for img in cropped_images_to_detect:
            # Template:
            cropped_img = cv.imread(img, cv.IMREAD_UNCHANGED)
            self.templates.append(cropped_img)
            # Sizes:
            cropped_w = cropped_img.shape[1]
            cropped_h = cropped_img.shape[0]
            self.template_sizes.append((cropped_w, cropped_h))
    # Detect cropped images on the main image:
    def detect_cropped_image(self, main_img, threshold, method=cv.TM_CCOEFF_NORMED):
        self.main_img = main_img
        # Generate results for each template:
        self.rectangles = []
        for i in range(len(self.templates)):
            # Get detected cropped image locations on the main image:
            result = cv.matchTemplate(self.main_img, self.templates[i], method)
            locations = np.where(result >= threshold)
            # Zip the detected cropped image locations into (X, Y) position tuples:
            locations = list(zip(*locations[::-1]))
            # Create the rectangles list [x, y, w, h] to avert overlapping positions:
            for loc in locations:
                w, h = self.template_sizes[i]
                rect = [int(loc[0]), int(loc[1]), int(w), int(h)]
                self.rectangles.append(rect)
                # For single points:
                self.rectangles.append(rect)
        # Group the rectangles list:
        self.rectangles, weights = cv.groupRectangles(self.rectangles, 1, 0.7)
    # Draw rectangles or markers on the detected croppped images:
    def get_and_draw_points(self, result_type, rect_color=(255,0,255), rect_type=cv.LINE_4, rect_thickness=3, marker_color=(255,0,255), marker_type=cv.MARKER_STAR, marker_size=25, marker_thickness=3):
        self.detected_center_points = []
        if(len(self.rectangles)):
            # Loop all detected image locations in the rectangles list:
            for (x, y, w, h) in self.rectangles:
                # Define the rectangle box points:
                top_left = (x, y)
                bottom_right = (x + w, y + h)
                # Define the rectangle center points for drawing markers:
                center_x = x + int(w/2)
                center_y = y + int(h/2)
                # Save the center points:
                self.detected_center_points.append((center_x, center_y))
                # Draw results on the main image:
                if(result_type == 'boxes'):
                    # Draw rectangles around the detected cropped images:
                    cv.rectangle(self.main_img, top_left, bottom_right, color=rect_color, thickness=rect_thickness, lineType=rect_type)
                elif(result_type == 'markers'):
                    # Draw markers on the detected cropped images:
                    cv.drawMarker(self.main_img, (center_x, center_y), marker_color, marker_type, marker_size, marker_thickness)
        return self.main_img
        # Clear the rectangles list to avoid errors:
        self.rectangles = []
    # Get the center points of the detected cropped images:
    def get_center_points(self):
        return self.detected_center_points


arduino_serial_comm.py

Download



import json
import serial
from time import sleep

def arduino_serial_comm(port, character):
    # Define the data file (json):
    json_file = 'assets/villagers.json'
    # Define the build-in Arduino Leonardo port on the LattePanda:
    arduino = serial.Serial(port, 9600, timeout=1)
    # Elicit information:
    with open (json_file) as data:
        villagers = json.load(data)
        c = villagers[character]
        msg = "Character: {}\n\nBirthday: {}\n\nLoves: {}\n\nLikes: {}\n\nHates: {}".format(character, c['birthday'], c['loves'], c['likes'], c['hates'])
        print(msg)
        arduino.write(msg.encode())
        sleep(1)
        


pics.py

Download



character_pics = {
        'Abigail' : ['assets/abigail_1.jpg', 'assets/abigail_2.jpg', 'assets/abigail_3.jpg'],
        'Emily' : ['assets/emily_1.jpg', 'assets/emily_2.jpg'],
        'Haley' : ['assets/haley_1.jpg', 'assets/haley_2.jpg', 'assets/haley_3.jpg'],
        'Maru' : ['assets/maru_1.jpg', 'assets/maru_2.jpg', 'assets/maru_3.jpg'],
        'Penny' : ['assets/penny_1.jpg', 'assets/penny_2.jpg', 'assets/penny_3.jpg'],
        'Alex' : ['assets/alex_1.jpg', 'assets/alex_2.jpg'],
        'Harvey' : ['assets/harvey_1.jpg', 'assets/harvey_2.jpg'],
        'Sam' : ['assets/sam_1.jpg', 'assets/sam_2.jpg', 'assets/sam_3.jpg']
    }


lattepanda_detection_results_display.ino

Download



         /////////////////////////////////////////////  
        // Stardew Valley Villager Recognition and //
       //      Gift Preferences Bot w/ OpenCV     //
      //             ---------------             //
     //         (LattePanda Alpha 864s)         //           
    //             by Kutluhan Aktar           // 
   //                                         //
  /////////////////////////////////////////////

//
// Via OpenCV, recognize villagers to display their birthdays, loves, likes, and hates while playing Stardew Valley on LattePanda Alpha.
//
// For more information:
// https://www.theamplituhedron.com/projects/Stardew_Valley_Villager_Recognition_and_Gift_Preferences_Bot_w_OpenCV
//
//
// Connections
// LattePanda Alpha 864s (Arduino Leonardo) :  
//                                3.5'' 320x480 TFT LCD Touch Screen (ILI9488)
// D10 --------------------------- CS 
// D8  --------------------------- RESET 
// D9  --------------------------- D/C
// MOSI -------------------------- SDI 
// SCK --------------------------- SCK 
// MISO -------------------------- SDO(MISO) 
//                                Buzzer
// D11 --------------------------- S     
//                                5mm Green LED
// D12 --------------------------- S


// Include the required libraries:
#include "SPI.h"
#include <Adafruit_GFX.h>
#include <ILI9488.h>

// Define the required pins for the 320x480 TFT LCD Touch Screen (ILI9488)
#define TFT_DC 9
#define TFT_CS 10
#define TFT_RST 8

// Use hardware SPI (on Leonardo, SCK, MISO, MOSI) and the above for DC/CS/RST
ILI9488 tft = ILI9488(TFT_CS, TFT_DC, TFT_RST);

// Define the buzzer pin:
#define buzzer 11
// Define the green LED pin:
#define green_LED 12

// Define the data holders:
String gameData = "";

void setup() {
  pinMode(green_LED, OUTPUT);
  // Initialize the serial communication:
  Serial.begin(9600);
  // Initialize the TFT LCD Touch Screen (ILI9488):
  tft.begin();
  tft.setRotation(1);
  tft.fillScreen(ILI9488_RED);
  tft.setCursor(0, 0);
  tft.setTextColor(ILI9488_BLACK); tft.setTextSize(4);
  tft.println("\nStardew Valley\n");
  tft.println("Villager Recognition\n");
  tft.println("and Gift Preferences\n");
  tft.println("Bot w/ OpenCV\n");

}

void loop() {
  // Via the serial communication, elicit the detected character's information:
  while (Serial.available()) {
    char c = (char)Serial.read();
    gameData += c;
  }
  // Display the transferred character information:
  if(gameData != ""){
    // Notify the user:
    tft.fillScreen(ILI9488_BLACK);
    tft.setCursor(10, 5);
    tft.setTextColor(ILI9488_RED); tft.setTextSize(4);
    tft.println("Stardew Valley\n");
    tft.setTextColor(ILI9488_GREEN); tft.setTextSize(2);
    tft.println(gameData);
    tone(buzzer, 500);
    digitalWrite(green_LED, HIGH);
    delay(500);
    noTone(buzzer);
    digitalWrite(green_LED, LOW);
  }
  // Clear:
  delay(250);
  gameData = "";
}


villagers.json

Download



{
	"Abigail": {
		"birthday" : "Fall 13",
		"loves" : "Amethyst, Banana Pudding, Blackberry Cobbler, Chocolate Cake, Pufferfish, Pumpkin, Spicy Eel",
		"likes" : "Quartz",
		"hates" : "Clay, Holly",
		"introduction" : "Abigail is a villager who lives at Pierre's General Store in Pelican Town. She is one of the twelve characters available to marry. "
	},
	
	"Emily": {
		"birthday" : "Spring 27",
		"loves" : "Amethyst, Aquamarine, Cloth, Emerald, Jade, Ruby, Survival Burger, Topaz, Wool",
		"likes" : "Daffodil, Quartz",
		"hates" : "Fish Taco, Holly, Maki Roll, Salmon Dinner, Sashimi",
		"introduction" : "Emily is a villager who lives in Pelican Town. She is one of the twelve characters available to marry. Her home is south of the town square, right next to Jodi's house, at the address 2 Willow Lane. She works most evenings at The Stardrop Saloon starting at about 4:00 PM."
	},
	
	"Haley": {
		"birthday" : "Spring 14",
		"loves" : "Coconut, Fruit Salad, Pink Cake, Sunflower",
		"likes" : "Daffodil",
		"hates" : "All Fish, Clay, Prismatic Shard, Wild Horseradish",
		"introduction" : "Haley is a villager who lives in Pelican Town. She's one of the twelve characters available to marry."
	},
	
	"Maru": {
		"birthday" : "Summer 10",
		"loves" : "Battery Pack, Cauliflower, Cheese Cauliflower, Diamond, Gold Bar, Iridium Bar, Miner's Treat, Pepper Poppers, Radioactive Bar, Rhubarb Pie, Strawberry",
		"likes" : "Copper Bar, Iron Bar, Oak Resin, Pine Tar, Quartz, Radioactive Ore",
		"hates" : "Holly, Honey, Pickles, Snow Yam, Truffle",
		"introduction" : "Maru is a villager who lives in The Mountains north of Pelican Town. She's one of the twelve characters available to marry. She lives north of town with her family in a house attached to Robin's carpenter shop. In the Social Status menu, Maru's outfit will change to a nursing uniform when she is at her job at the clinic."
	},
	
	"Penny": {
		"birthday" : "Fall 02",
		"loves" : "Diamond, Emerald, Melon, Poppy, Poppyseed Muffin, Red Plate, Roots Platter, Sandfish, Tom Kha Soup",
		"likes" : "All Artifacts, All Milk, Dandelion, Leek",
		"hates" : "Beer, Grape, Holly, Hops, Mead, Pale Ale, Pina Colada, Rabbits Foot, Wine",
		"introduction" : "Penny is a villager who lives in Pelican Town. She's one of the twelve characters available to marry. Her trailer is just east of the center of town, west of the river. She teaches Vincent and Jas."
	},

	"Alex": {
		"birthday" : "Summer 13",
		"loves" : "Complete Breakfast, Salmon Dinner",
		"likes" : "All Eggs (except Void Egg)",
		"hates" : "Holly, Quartz",
		"introduction" : "Alex is a villager who lives in the house southeast of Pierre's General Store. Alex is one of the twelve characters available to marry."
	},
	
	"Harvey": {
		"birthday" : "Winter 14",
		"loves" : "Coffee, Pickles, Super Meal, Truffle Oil, Wine",
		"likes" : "Daffodil, Dandelion, Duck Egg, Duck Feather, Ginger, Goat Milk, Hazelnut, Holly, Large Goat Milk, Leek, Quartz, Snow Yam, Spring Onion, Wild Horseradish, Winter Root",
		"hates" : "Coral, Nautilus Shell, Rainbow Shell, Salmonberry, Spice Berry",
		"introduction" : "Harvey is a villager who lives in Pelican Town. He runs the town's medical clinic and is passionate about the health of the townsfolk. He's one of the twelve characters available to marry."
	},
	
	"Sam": {
		"birthday" : "Summer 17",
		"loves" : "Cactus Fruit, Maple Bar, Pizza, Tigerseye",
		"likes" : "All Eggs (except Void Egg), Joja Cola",
		"hates" : "Coal, Copper Bar, Duck Mayonnaise, Gold Bar, Gold Ore, Iridium Bar, Iridium Ore, Iron Bar, Mayonnaise, Pickles, Refined Quartz",
		"introduction" : "Sam is a villager who lives in Pelican Town. He's one of the twelve characters available to marry. He lives in the southern part of town, just north of the river at 1 Willow Lane. Sam is an outgoing and energetic young man with a passion for music. He works part-time at JojaMart."
	}
}

Schematics

project-image
Schematic - 69.1


project-image
Schematic - 69.2

Downloads

Fritzing

Download


Stardew_Valley_Vision.zip

Download