Building in Public: How to determine what features to build with limited information
Making decisions with imperfect information, competitor usage analysis, knowing when to fold, deserving success
Like most things in life, being a startup founder is about tradeoffs. With highly constrained resources — specifically time, capital and engineering resources — one of the most critical responsibilities is resource allocation.
Yet, this presents a conundrum. At the moment when resource allocation is most constrained, the information available to one to make a decision is most limited. In many ways, it’s like trying to win the World Series of Poker with a fraction of the chips and only knowing a single card in your hand.
Hence, the high failure rate. It’s hard to win that game because the game is not designed to be won that way.
Thus, it becomes the job of the CEO and founder to either gain more chips (acquiring more resources) or figure out how to play your hand. Yet, for all but a select few, it’s hard to get more chips without proving you know how to play the game and win some hands.
So what do you do?
First, you must understand the dynamics of the game you are playing and what information is (1) accessible and (2) valuable.
In Texas Hold Em, it would be valuable to know what other cards your opponents have, but that information is not accessible. Conversely, it is quite accessible to know the community cards dealt during the flop (the first three cards played by the dealer face up), but if you don’t know that a straight beats a two pair you might fold a winning hand.
Yet, there is more nuance here. Information has transactions costs. User research, for example, is sales. It requires hours prospecting, outreaching and then, hopefully, interviewing. It also requires preparation to understand what information from a given user research sprint is valuable — as refined in specific hypotheses — and what questions and interview patterns unearth that information with the least amount of bias.
Over the past month, I conducted somewhere between 50+ user research interviews for my startup. As we are undergoing the natural pivot, I will keep the user segment and nature of these interviews private. But they consisted of three specific personas that required their own specific outreach strategies. The easiest segment allowed me to go direct. The second hardest segment required me to leverage Instagram’s social graphs. The hardest segment required me to scrape the Instagram like history of supply in a certain vertical then manually review 5,000+ profiles to compile follower information and potential email contacts. From there, I conducted hundreds of cold email outreach with more emails ignored than not. Alas, transaction costs.
So, when transaction costs are high and information and resources are constrained, grunt work and creativity become extremely valuable skillsets. Creativity is an important attribute as it suggest a propensity to think-outside-the-box. This helps identify unconventional solutions to a given problem: in this case possible channels and resources from which one could procure valuable information that is highly accessible.
The following analysis is one such example.
I am sharing this work to provide documentation of my decision-making process as a founder and leader for future investors and employees. I also hope that other entrepreneurs and would-be-founders find this process as a call to action to embrace the grunt work and get creative when making decision with imperfect information.
This analysis, combined with user research and certain MVP test data, gave us the conviction to fold a hand at the moment that the bet was raised. This, in hindset, was a good decision. More importantly, our decision making process was sound.
The role of a startup leader is not simply resource allocation. Communication is vital. How you communicate internally how certain decisions were made and why is critical to creating alignment.
I find writing as the most effective tool for not just synthesizing imperfect information to arrive to a decision, but also as a mechanism to explain how we arrived their and why it matters.
Boats move faster when the crew paddles in the same direction. In the high seas of startups, the water can get choppy and the wind fierce. When communication is difficult, write.
Overview
This analysis seeks to understand how different products features are used by community users. Specifically, we will explore the relative value creation of different product features by measuring post frequency by feature. We will also measure the value that certain features create for the community as mediums of interaction based on the average likes per post and comments per post by feature type. We will focus on three communities that have proven product-market fit in a category of interest (beauty), they are:
Glossier (Reddit)
Jones Road Beauty (Facebook)
Beauty Pie (Facebook)
And one community (The Daily Planner) from Mighty Networks that lacks the scale of the above communities but has signs of durable engagement.
Research Question
What features create the most value across top performing communities and are there areas where users needs are being undermet?
Methodology
This analysis seeks to answer our research question across four independent sample sizes of 100 posts from the four referenced community. Jones Road Beauty, for example, had 663 posts over in the last month. We believe the number of posts over a 30 day period is the right population size as it is exhaustive enough to capture all community use cases and the usage of product features to best meet them. Our sample size of 100 implies a 95% confidence interval with a 9% margin of error.
For each post, we recorded the type of product feature used, the number of likes and the number of comments.
For Glossier’s Reddit Community we also recorded the post’s flair: Reddit’s native theme tagging feature.
For Jones Road Beauty’s Facebook Community, we also recorded the post creator’s ‘badge’, the emotion expressed by the post and the post content type by use case.
For Beauty Pie’s Facebook Community, we recorded the post creator’s ‘badge’ and post content type by use case.
Goal
The goal of this analysis is to evaluate which features are currently being used in existing communities, especially in context of (1) the specific use case and (2) what other features are available.
This will allow us to better understand what existing (or non-existing) features met user needs for certain use cases and expressions. By doing so, we can determine, prioritize and potentially kill certain elements of our roadmap.
Analysis
Reddit
Reddit provides a relatively wide feature set for their content. These include:
Text-only Posts
Image-focused Posts
Videos
AMAs
Polls
Discussion Boards / Moderator Threads
Brand Announcements
Glossier
In Glossier’s 42k member Reddit community, users solely opted for Text-only posts and Image-focused posts form factors across the various use cases employed by the community. Image-focused posts had nearly an equivalent usage as Text-only posts. Interestingly, users did not employ Polls in use cases where one might imagine it to be an effective feature. A single post employed a Moderator Thread / Discussion Board.
Moreover, Image-focused posts had 5.8x greater average like per post and nearly the same number of comments as Text-only posts. This implies that this feature (well rendered images with a header) is both a popular medium for creators to express themselves and creates more value to the community as they reward the creator with upvotes and provide them with the requested feedback or support outlined in the post via comments.
Additionally, all posts in Glossier’s Reddit community require the creator to include a “flair” which adds as a “tag” to denote the topic of the post. The Flairs are:
Collection/haul
Discussion
Questions
Issues
Tips
News
Other
Review
Swatches
Furthermore, we analyzed the feature frequency by flair. By doing so, we can evaluate and rank community use cases and understand which product feature best meets the Job to be Done of the user to express a certain use case.
Similarly, we analyzed average likes and comments per post across different flairs. This allowed us to observe which feature provided more relative value to the community in different use cases.
Some use cases, such as Questions, are more frequently expressed in text (86% of the time). But in the circumstances that they are expressed in image-focused posts, it creates significantly more value for the community who compensate the creator with 12x more likes and 2.4x more comments on average. Interestingly, community members leveraged image-focused posts 42% of the time for Discussions — the most common use case found in the community. When they did so, they created significantly more value on average for the community (6.2x more likes) and fostered slightly more responses on average, which ultimately is the goal of a ‘discussion’.
Actionable Insight
Users do not value — or do not know that they value* — polls as a feature to express the different use cases across community.
*This may be a function of product design and UI.
*If further research disproves this conclusion and informs that users find polls valuable, then the following lessons from Reddit’s UI should be severely weighted in any UI design.
Reddit’s posting UI offers three features at the top of the feed for users to quickly employ to create a post: text, image and link. As we have seen, this UI has a direct impact on which features users’ employ.
Upon clicking any of those options, a user is sent to a new view where they complete the remaining actions for the specific feature and select a relevant flair. It is in this view that a user now sees the option to create a poll. This creates added friction and confusion.
Images create more value for the community and are used across many use cases; we should attempt to make image-focused posts the default
Our product design should make the act of uploading or posting an image extremely easy and the default action should permit sharing image-focused posts as easy as sharing text-only posts. This means the original call to action to create content should include an action relating to sharing images.
Moving on, both Jones Road Beauty and Beauty Pie host their communities in Faceook Groups.
Facebook Groups
Facebook Groups credentials users with badges. Facebook’s badge system is determined based on the accumulation of Contributions. Users can have multiple badges.
Facebook FAQs
What are Contributions?
Contributions reward you for being a great contributor in your group! As you participate in your group, you’ll be eligible to earn different badges.
What’s the value of holding a badge?
When you get a badge, others will be able to recognize you as a top group contributor! It shows you regularly contribute to the group by creating posts or leaving comments.
Do these carry over to my other groups?
No, each of your badges are specific to one group.
Facebook Groups also provide a wide range of features. These include:
Text-only Posts
Image-focused Posts
Featured Posts
Polls
Videos
Rooms
Feeling/activity
Check in
Create Event
Tag event
Tag people
Livestreams
Reshares
Embedded Media
Nonetheless, users have willed certain “features” into existence with their own makeshift versions that are a clunky repurposing of links, photos and text. These include:
Clunky Product Tagged Photos
Shoppable Product Launches
Expert Consultation
Product development Input and Feedback
Direct Messaging the Brand
External Discounts
Gifting/Resale
Competition/Challenge
Shoutouts
Screenshot (PDP Image, IG Post)
Jones Road Beauty
Jones Road Beauty’s 18.2k member Facebook Community used a range of product features with text-only being the most common. Importantly, user-created “clunky Tagged Product Photos” was the second most common feature used for content creation. Clearly, users gained value from sharing a photo — most commonly a selfie of themselves — and ‘tagging’ the different products that they were wearing. But given the lack of tooling for this feature, users did so in plain text by ‘tagging’ the products in the post’s caption.
Again, polls are not being employed as a feature across all the different use cases for content creation in the community. Moreover, Facebook’s UX is designed to push users to create ‘Rooms’. However, this product is seldom used and does not appear to create any value for the community.
As we saw with Glossier’s Reddit Community, image-focused posts appear to create more value for the community. Image-Focused Posts had 5.5x more likes and Clunky Product Tagged Photos had 4.7x more likes than text-only posts. Moreover, Imaged-focused Posts had 1.7x more comments and Clunky Product Tagged Photos had 1.5x more comments than Text-only Posts. This implies that this feature creates more value to the community as a means of consumption (rewarding the creator with likes) and social interaction/engagement (rewarding the creator by spending the time and energy to provide them with the requested response).
Moving on, creators of posts in this community ranged in all types of engagement, which was measured by their badges. The most active contributor were individuals with the Top Contributor badge, followed by New Members and users with no badge.
While Top Contributors, New Members and users with no Badge post the most in JRB community, they create the least amount of value for the community on a per post basis as measured by likes per post and comments per post. These users use text-only post features the most frequently.
Admin, Moderators, Group Experts, Visual Storytellers and Rising Stars create a lot of value for the community but do not post frequently. These users use text-only post features the least frequently. Instead, specifically in the case of Admins and Visual Storytellers, they often employ ‘user-created’ product features like Clunky Product Tagged Photos. These posts, in addition to Embedded External Media and Reshared Posts, appear to create more value for the community than by which they are produced.
What emotions were users most commonly expressing when sharing posts and interacting with the community?
Users most commonly created posts in the community and sought out interaction to share their gratitude of JRB and the happiness and contentment it has given them in their life. Often, they were curious about products and what to buy given certain use cases and conditions, as well as what the community thought or recommended regarding how they can improve their Job to Be Done.
The community most valued posts where users shared vulnerability, excitement, confidence, happiness, enthusiasm and gratitude. The community less valued posts that expressed uncertainty, frustration and curiosity, although they provided support in the form of comments.
Negative emotion posts were most commonly shared via text-only product posts whereas the valued emotional sentiment outlined above leveraged imaged-based posts, specifically Clunky Product Tag Photos.
We see a similar distribution of usage across the different use cases and content types, which we discusses in Online Communities: What Makes Them Tick?.
Actionable Insights
Product Tagged Photos appear to be the most actionable feature to build.
As we saw with the other communities, image-focused posts appear to create more value for the community. This implies that this feature creates more value to the community as a means of consumption (rewarding the creator with likes) and social interaction/engagement (rewarding the creator by spending the time and energy to provide them with the requested response).
Reshared Posts, Embedded External Media, Shoppable Product Launches and Videos create a lot of value and engagement from the community but are not used frequently by post creators as a means to express certain community use cases.
Polls are never used.
While Facebook’s UX pushes users to create Rooms, it does not appear to have product-market-fit with the use cases of the community and is rarely used.
Top Contributors, New Members and users with no Badge post the most in JRB community but create the least amount of value for the community on a per post basis as measured by likes per post and comments per post
Admin, Moderators, Group Experts Visual Storytellers and Rising Stars create a lot of value for the community but do not post frequently. Product features and UI should be designed to get these users to post more. One way to do so is by investing in the feature sets that they are duct-taping together.
Some potential features that could best serve these users include:
Shoutouts
Reshares
Embedded Media
Direct Messaging
Livestream
Integrated Shoppable Product Launches
Expert Tooling
Color Recommendation and Selife Submissions
Source; Facebook Tickets and events
Product Development Input and Feedback Requests
Conversation Starters post at a high frequency but appear to be creating less relative value for the community and not spurring as much conversation as other user types (as measured by average number of comments per post).
Embedded External Media and Reshared Posts appear to create more value for the community than by which they are produced.
The JRB community most valued posts where users shared vulnerability, excitement, confidence, happiness, enthusiasm and gratitude.
Negative emotion posts were most commonly shared via text-only product posts whereas the valued emotional sentiment outlined above leveraged imaged-based posts, specifically Clunky Product Tag Photos.
Facebook’s Posting UI pushes Rooms, People Tags, Check Ins and Feeling/activity — product features that are never used and are more suited for general Facebook usage not community usage. Similar to the issue with Reddit, users do not find value in polls — or do not know that they find value in polls given how buried they are in the user journey to create a post.
As a result, polls are not being used in obvious use cases.
Beauty Pie
Beauty Pie’s 24.4k member Facebook Community used a range of product features with text-only being the most common. Compared to JRB, users create less “Clunky Tagged Product Photos”, but instead use images in other use cases. BP users have created their own feature in the form of ‘screenshotting’ to best meet a specific use case relating to getting information at specific points in the customer journey while they shop on site.
Again, polls are not being employed as a feature across all the different use cases for content creation in the community. Moreover, Facebook’s UX is designed to push users to create ‘Rooms’. However, this product is seldom used and does not appear to create any value for the community.
As we saw with the other communities, image-focused posts appear to create more value for the community.
Compared to JRB, users create less “Clunky Tagged Product Photos”, but instead use images in other use cases. BP users have created their own feature in the form of ‘screenshotting’ to best meet a specific use case relating to getting information at specific points in the customer journey while they shop on site.
Moving on, Users with no Badge post the the vast amount of content in the BP community, followed by Conversation Starters, New Members and Visual Storytellers. This community has much less diverse user base by contribution and do not have many power users actively contributing.
Visual storytellers and users with No Badge create the most value to the community as measured by both likes and comments.
Image-based product such as tagged photos, image-focused posts and screenshots create the most value for the community and are most commonly used by user types that create the most value for the community.
Actionable Insights
Image-based product such as tagged photos, image-focused posts, screenshots create the most value for the community and are most commonly used by user types that create the most value for the community.
Compared to JRB, users create less “Clunky Tagged Product Photos”, but instead use images in other use cases. BP users have created their own feature in the form of ‘screenshotting’ to best meet a specific use case relating to getting information at specific points in the customer journey while they shop on site.
Users with no Badge post the the vast amount of content in the BP community, followed by Conversation Starters, New Members and Visual Storytellers.
The Community suffers from the lack of posts from Admin, Moderators and Group Experts, who create a lot of value for community members.
Admins ‘user-create’ brand announcements. These posts created the most value to the community and saw strong comment contribution. However, the lack of feature set for this use case seems to have created friction that likely resulted in less content creation and community value as a result. Features focused on getting Admins, Moderators, Group Experts and Rising Stars should be prioritized.
Beauty Pie Community members currently take a snapshot of the PDP and then share that image to the Facebook community to get feedback because that’s where individuals they trust reside. This creates friction and takes the user away from the store. Nonetheless, this use case and hacked together feature drives conversation in the form of comments. This feature is used to express a number of use cases in different parts of the customer journey.
How do shoppers leverage brand communities during the customer journey?
Interactions on PDP
Product Feedback to help inform a buying decision even when reviews are available
Requesting specific feedback from community regarding product X given condition/question Q
Seeking information and support for a specific problem
Requesting support and advice from the community regarding problem P that user is facing with product X
Seeking information and advice from the community on how to better use a product
Requesting information from the community regarding how to effectively use product X given condition Q in order to improve its performance for use case Z
Product review
Sharing feedback with the community regarding how product X has performed on use case Z given condition Q or comparison to product Y
Complaint
Sharing a complaint with the community regarding how issue S has made product X underperform for use case Z
By eliminating the need to screenshot and instead creating a feature that enables the creator to reference a certain product and distribute natively alongside the customer journey, we can increase the value that these posts provide for the creator by making them more convenient.
Interactions at Checkout
Three questions as I “load up my JR cart” for travel friendly items
Product Comparison for specific use case
Requesting comparative feedback from the community regarding how product X (in cart) compares to similar product Y for use case Z
Cleansing stick (X) - do you like it equal to or better than micellar water (Y) for makeup removal (Z)?
Product Feedback for specific use case given certain constraints
Requesting specific feedback from the community regarding how product X (in cart) performs for use case Z given condition Q
Oil stick (X) - has anyone with an oily Or combo complexion (Q) tried it on their roscea or flaky spots (Z). Don’t want to make pours worse but always looking for things to help
Product Variation Feedback for specific use case
Requesting specific feedback regarding specific variations of product X (X1, X2, X3, etc.) for use case Z given condition Q
Shimmer oil - Midas (X1) or Bronze (X2) for a redish face (Z)? I love (loooove) golden hour MB and use bronze MB for a little contour. But I’m confident I will not be able to handle MB with my natural oils and humidity (Q). Already have opal SFO and loved it in winter.
Input on how product can improve
Providing specific input regarding how product X can be improved and how the brand can better address use case Z given condition Q
Input for Sydney… it would be interesting if there could be a lighter weight version of WTF (X), like the light weight cream vs miracle cream. Moving into summer Id like the coverage that WTF does provide in smoothing about my color (Z), but don’t need the extra moisture (Q).
Mighty Networks
Mighty Networks provides a wide range of features. These include:
Text-only Posts
Image-focused Posts
Featured Posts
Polls
Hot and Cold
Questions
GIFs
Introductions
Embedded Media
Videos
Rooms
Full-screen Image
Nonetheless, like we have seen elsewhere, communities leveraging MN have willed certain “features” into existence with their own makeshift versions that are a clunky repurposing of links, photos and text. These include:
Clunky Product Launch
Clunky Product Tag Photos
Challenges
Product Input
Screen Recording
Gifting/Selling
Brand Announcements
The Daily Planner is a community that recently transitioned from Peerboard to Mighty Networks.
This community used a range of product features with Image-focused Posts being the most common. Interestingly, text-only posts were quite rarely used in comparison to other communities. The third most common product feature used was questions, which was often used instead of polls to gather feedback and information from the community as part of a users Job to be Done.
Similar to to other communities, we saw users will certain features into existence, specifically clunky product launches that tried to embed commerce and shopping instances within the community, as well as clunky product tagged photos and challenges.
Again, polls and hot cold are infrequently employed as a feature across all the different use cases for content creation in the community.
As we saw with the other communities, image-focused posts appear to create more value for the community. Gifs and embedded external media often conveyed emotion and lightheartedness that entertained and please the community which saw the most amount of likes. Users also gained value from posts requesting and providing product input (not using the polls feature) and found value in the common connection involved in challenges.
Questions by and far received the most median responses from the community. Image-focused posts, GIFs and product input use cases saw high engagement in terms of the community commenting on posts that employ these features.
The Admin or Host was an extremely active contributor in the community and saw the most likes and comments per post.
The Admin/Host predominantly employed the features that created the most value for the community, including:
Embedded External Media
GIFs
Product Input
Challenges
Interestingly, Questions were used across user types.
Features for Admins
Aggregating our analysis, we can see the frequency by which these features were used by Admins across the different communities.
What we did with this analysis, unfortunately dear reader, I am keeping close to the vest.
Closing Thoughts
Being a founder, so I have learned and come to love, is about doing the work. It’s about trusting the process, as my friend Sam would say. As my team knows, and often hears me say, production makes proficiency. I wrote about this a bit in What Makes IP Valuable. What this means is that you don’t become Michael Jordan without first putting in the hours practicing, training, working out, failing, falling, hurting, and what not. That work — that production — is worthless to everyone but him, because he gains value from the process of getting better. 99% of your time as an entrepreneur is spent doing work that’s main source of value is internalized not externalized.
Everyone has a will to win but very few have a will to prepare to win.
- Vince Lombardi
One of our values at Banter is Hone Your Craft. You hone your craft when know one is watching, working on things that no one will pay for, arriving at results that nobody ever sees. But that’s how you compound a skillset and a will to prepare to win.
This, perhaps more than anything else, is the most important trait that we look for in potential hires. A relatively controversial question that I frequently ask in interviews is, “why do you deserve to be successful?”.
It is blunt, but I find it effective. All humans want, and so wanting to be successful is an extremely easy thing to say and does not psychological threaten our sense of integrity.
Deserving, I have found, is an interesting filter. For one to say they deserve something, it implies that they have already born some cost and accomplished something worthy — whether a quality or an achievement — to arrive at that moment in time. When we say we deserve something, it intuitively challenges our integrity because it is a function of where we have been rather than where we hope to go. We are green when we want, we are hardened when we deserve. It is much harder to fake and even more importantly, vibrantly shines through when genuine.
Why do you deserve to be successful?
