Midscene.js

Joyful UI Automation

Midscene.js is an AI-powered automation SDK can control the page, perform assertions, and extract data in JSON format using natural language.

Features ✨

• Natural Language Interaction 👆: Describe the steps, and let Midscene plan and control the user interface for you
• Understand UI, Answer in JSON 🔍: Provide prompts regarding the desired data format, and then receive the expected response in JSON format.
• Intuitive Assertion 🤔: Make assertions in natural language; it’s all based on AI understanding.
• Experience by Chrome Extension 🖥️: Start immediately with the Chrome Extension. No code is needed while exploring.
• Visualized Report 🎞️: With our visualized report file, you can easily understand and debug the whole process.
• Out-of-box LLM 🪓: It is fine to use public multimodal LLMs like GPT-4o. There is no need for any custom training.
• Totally Open Source! 🔥: Experience a whole new world of automation development. Enjoy!

Resources 📄

• Home Page: https://midscenejs.com
• Quick Experience By Chrome Extension, this is where you should get started
• Integration
  • Automate with Scripts in YAML, use this if you prefer to write YAML file instead of code
  • Bridge Mode by Chrome Extension, use this to control the desktop Chrome by scripts
  • Integrate with Puppeteer
  • Integrate with Playwright
• API Reference
• Customize Model and Provider（e.g. using Gemini）, see how to use your own model and provider

Community

• Discord
• Follow us on X
• Lark Group

License

Midscene.js is MIT licensed.