User Testing Your Shopify App: Public App Use Cases You Should Test

User Testing Your Shopify App: Public App Use Cases You Should Test
user test app

So you’re building a Shopify App—that’s amazing! Apps are a critical part of the Shopify ecosystem and fuel countless merchant stores every day.

Building a Shopify App is the mark of a great developer, because not only are you building an app for a merchant, you’re building an app for the merchant’s customers as well.

Taking into account the complexities of Shopify with the dynamic landscape of commerce can be pretty tricky. You might think you’ve completed the project after finishing that last line of code— but really, your journey is just beginning.

Become a Shopify App Developer

Join the Shopify Partner Program and power the world's entrepreneurs. When you sign up, you'll get a free API key, so you can build for 600,000 businesses transforming the world of commerce.

Sign up

User testing your app

Before you start shipping your app and submitting it to the Shopify App Store or to merchants directly, you need to remember to thoroughly test your app. On the App Quality Assurance team at Shopify, we see a great number of in-beta or incomplete initial app submissions. While we appreciate the enthusiasm and eagerness, you do need to ensure you create feature-complete products that are ready for our eyes and merchant eyes.

This means that you have to test your app as thoroughly as possible, because not only does this app impact your business; it impacts merchants’ businesses as well.

In theory, testing sounds pretty simple. You build an app, test installation, verify the main functionality of the app, and you’re good to go, right? Wrong! Shopify is a complex platform, and every one of our merchants is unique. So while you will build your app for the many, there are some particular use cases that will come up that you should be testing, too.

Pressure is on, but not to worry—in this article, we share a quick list of testing scenarios. This information is designed to give you necessary use-case context within the Shopify ecosystem. It’s also here to ensure your app looks great for merchants, and will help you pass the review process when you submit your app.

Throughout the post, we’ll get into some common use cases and some basic ground to cover including core online store components, installation, products, orders, billing and general principles. Let’s jump in!

Throughout this post, you will be asked to test using dummy shop data. A great way to generate this in bulk is to use the Developer Tools app.

Testing the online store

If you’re building an app that impacts the online store, you are in good company. For many merchants, the online store is their main or only sales channel. Being such, when building an app for the online store, you must ensure you are building a product that will make the main business channel look good.

user test app: app store
The store design section of the Shopify App Store.

Some important things to consider here are:

  • Impact on the cart: The cart is how merchants’ customers reach checkout, which means it’s your job to ensure merchants don’t miss a sale. If your app impacts the cart in any way, you have to make sure that you test it across multiple themes, with multiple apps (using partner friendly apps), and with multiple different product combinations in the cart. Page and drawer type carts should both be tested. When in doubt, have a contingency plan and ensure merchants have all necessary setup instructions, uninstall instructions, app limitations, and requirements.
user test app: theme editor
Cart drawer settings in the Theme Editor .
  • Shop domain: While some merchants do wish to keep their initial .myshopify domain, many merchants do not and opt for custom domains. Your app should be able to handle the merchant’s correct shop information and brand the shop correctly. 
    • Mobile responsiveness: It’s 2019… that is all. But in all seriousness, if you’re building an app that impacts the storefront, you should be building mobile first. More than half of all online transactions happen through mobile, so your app must be as mobile responsive as our merchants’ themes.
    • Multiple themes: So many apps produce errors on themes or don’t have expected outcomes on certain themes. While Shopify themes follow a similar pattern and have become more organized since they were sectioned off, all themes are still different. This is why as developers building for the storefront, you can’t rely on testing just one theme (such as Debut). You have to test with different themes. You can also fill in theme sections with text and placeholder images made easier through our Burst integration.
    user test app: themes
    Free Shopify themes accessible through the themes admin dashboard.

    Testing different order and product scenarios

    You can’t just test for starter shops. While many of the merchants who join the platform are fresh entrepreneurs looking to start their new businesses, that can’t be the only type of merchant your app serves. Shopify powers businesses of all different sizes, and your app should be tested to accommodate that.

    Some important things to consider here are:

    • Test on multiple locations: Shopify launched Locations in May 2018. Many merchants operate using inventory from different locations, and your app needs to be able to accomodate. 

    • Test on shops with sold out stock: I can’t tell you the number of times, as a reviewer, an app has tried to sell me a sold out product and allowed me to add it to cart, only to be rejected by the checkout. There are also many times that apps that work in the back end for merchants end up producing inventory or sales order issues for shops with sold out products. The best way to avoid this? Set up some sold out products and see how your app interacts with them.
    user test app: sold out
    Sold out product.
    • Test on shops that do not track inventory: Shopify is comprised of all different types of merchants, some of whom opt to not track inventory in Shopify. This could be anything from grocers that sell a variety of different foods based on weight, to a dropshipping merchant that doesn’t need to worry about inventory. Your app should be able to handle inventory that is not tracked as well as inventory that is. A key piece of advice: do not treat N/A stock as ‘out of stock’—it could cost a merchant a sale.
    • Test on products with ‘compare at prices’: Many merchants take advantage of compare at prices to make their sales all the more visible to customers and to also better document marked down items for reporting. Your app must be able to successfully handle the discrepancy and use the appropriate price where relevant.
    • Testing on product variants: Variety is the spice of life and that is ever apparent in commerce. When shopping you’ll often notice the same product is sold in various different sizes, colours, editions, etc. To support this, Shopify merchants take advantage of the Variants functionality, offering merchants a different set of properties for the different variations of their main product. You must take variants into consideration when initially building the app.
    user test app: variants
    Product in different size variants.
    • Process various test orders: If your app impacts any post checkout process (i.e. order status, inventory adjustments, post-purchase marketing, shipping and fulfillment), you need to be performing your due diligence as the app developer and performing multiple different test orders. These test orders should include controlled groups of app-affiliated resources, non app-affiliated resources, and a combination of both.
    • Test on different order statuses: Merchants use many different payment methods, from payment gateways to cryptocurrency to cash on demand. As a result of these different payment methods, not all orders will be marked ‘paid.’ The same can be said about fulfillment (i.e. fulfilled and unfulfilled). You must be mindful of these different payment and fulfillment statuses. If your app operates in fulfillment, either physical or digital, a crucial piece to test for is whether your app automatically fulfills pending orders (not paid). This could put merchants at a loss should payment fail and in doing such, hurts your business as well as the merchant’s.

    You might also like: How to be Successful on the New Shopify App Store.

    Testing installation

    Although I mentioned this as a simple step earlier on this article, you’d be surprised at the number of apps that get rejected per month based on failure to properly install according to our standards.

    Here are some key things to remember about install:

    • Cross check your code to your app set up info: In your Partner Dashboard under your app name and app setup, you’ll find some URLs and some token info. These are the URLs and tokens you have (hopefully) plugged in to your app’s code. If they’re not, your app will likely fail install or attempt to install another app to your test shop.
    • Test direct install from the Partner Dashboard: This will predominantly apply to those apps that want to submit to the Shopify App Store. It’s one thing to use a direct install URL and plug in your .myshopify domain, but it’s another to go directly to the OAuth screen when Shopify gives you the shop properties for install. To ensure your app is moving to the OAuth screen first after hitting Add app, test the install on a development store in your Partner Dashboard. More info on how to do this can be found in our docs
    user test app: install function
    Partner Dashboard development store direct app install function.
    • Double check your embedded app: Embedding your app in Shopify gives you a lot of great resources and the possibility to add new functionality. But embedded apps also breathe a lot more complexity into your process. You need to make sure that the content you load is hosted in the same place, and that all content is SSL secured, otherwise it will not display. You also need to make sure that your app is properly redirecting out to content if you do choose to display other content. Rule of thumb: click every single button and link in the embedded view.
    • Test reinstall: There are two main ways to test reinstall. You can test another install about a minute after you uninstall the app in the same session, or you can test reinstall after you’ve cleared your cache. Both of these scenarios are likely to happen, and your app needs to handle both.
    • Test the app on another shop with the same shop email: It should be no surprise to you at this point that Shopify merchants have different business models and structures, meaning some merchants choose to operate multiple shops under the same email. If they’ve fallen in with your app, it’s a good thing! To keep their affections, you need to test your app on multiple shops with the same email address to grant that merchant the functionality they need error free.

    Testing your billing

    If you’re building a public app you want to submit to the Shopify App Store, you are required to use the Billing API unless your app and service is completely free. In implementing the Billing API, you’re also required to test how your app is using it, including running through a test of accepting its charge. Learn more about how to do this in our docs.

    Here are some scenarios that you should test:

    • Upgrade, downgrade, upgrade, downgrade: Merchants’ businesses are sometimes seasonal, meaning there are high traffic points and low traffic points. Merchants may therefore require lower tier plans at some points in the year, and higher tier plans at others. You should test your app to make sure that it’s able to handle fluctuating plan selections.
    • Billing cancellations: I hate to be the bearer of bad news, but not everyone will accept your billing charge and as a result, they’ll hit cancel. As of December 2018, Shopify tells merchants when they have cancelled a charge. That doesn’t mean that your job is over. Merchants may revisit the app or try to choose another plan. You have to test your app to see not only whether cancelling a charge produces errors, but also if your app is able to POST another charge for the merchant to accept and for you to activate.
    user test app: cancelled charge
    Cancelled charge.
    • Check the Partner Dashboard after accepting a charge: If you’ve accepted a charge and are redirected back to the app’s user interface, that’s great. But you can’t forget to activate that charge in order to get paid. The simplest way to check activation is by looking through your Partner Dashboard app history.
    • Uninstall and reinstall: Merchants are business owners, but so are you. A common error I see is an app that forgets to charge merchants for a reinstall after the initial billing cycle is over. You need to get paid, so don’t forget to check this use case, because it is bound to come up in the future.

    You might also like: Getting Paid: An Overview of Shopify App Billing Cycles.

    General testing guidelines

    To sum up everything we’ve covered in this piece, below are some general guidelines to keep in mind when testing your Shopify App.

    • Test in a live environment: This is potentially one of the most important testing instructions and ties everything together. If you have a great app locally that breaks in production, you’ve failed. You need to test like your end user would. This means testing the in-production app, on a Shopify store, with some products, orders, customers, discounts and maybe a theme.
    • Test how your app handles errors: Users are unpredictable. User actions are unpredictable. If you’ve ever closed all your windows from hitting a button accidentally on your keyboard, you’ll know what I mean. This unpredictability needs to be taken into account, and you need to ensure you purposefully create errors to see how your app handles them. Try to access a 404, try to plug in an invalid param, try to access things that other users shouldn’t as a user.
    user test app: 404
    404 error that loaded in the embedded app.
    • Try to break your own app: All those sleepless nights you spent making sure all the code compiles has made you too close to this project. You know it inside and out, but you can’t let that stop you from testing critically. When too close to the project, consider asking a friend to try and break your app for you. Merchants are not the experts on your app and will be prone to making mistakes. When this happens, they may uninstall the app if they are too confused. Mitigating the potential downfalls of the app as much as possible and trying to break it before someone else does is a great way to future proof your app.

    Become a Shopify App Developer

    Join the Shopify Partner Program and power the world's entrepreneurs. When you sign up, you'll get a free API key, so you can build for 600,000 businesses transforming the world of commerce.

    Sign up

    Avoid the ‘wait and see’ mentality

    You only get one chance at a first impression with your app. Rather than shipping an unfinished or beta project that might land you a rejection with the App Quality Assurance team, or get you complaints and negative reviews once live, be very meticulous with your testing. The stronger your app is at the start, the higher your install retention rate will be.

    There’s something really rewarding about growing the businesses of merchants while growing your own business, and we want you to experience that feeling to the fullest. Like with any good business, you have to release a great product. This doesn’t happen by accident. But by combining your creativity, your development skills, and some hard work in testing your product, we know you’ll be successful.

    What does your app testing process look like? Let us know in the comments below!

    About the Author

    Carine Araujo is an App Quality Assurance Specialist for Shopify. When she's not reviewing apps or drafting app quality related content, you can find her illustrating nerd comic concepts or making coffee.

    Grow your business with the Shopify Partner Program

    Learn more