2022. 12. 22 | 3mins

After introducing Upstage's Recommendation AI Pack, there are some people who have increased their customers' purchase conversions by an average of +60% and up to +150%. Upstage's RecSys team and AI Challenges team, which led to an AI recommendation system contract with amazing performance through PoC with vertical commerce company 'Brandi'!

In addition to the story behind the development of recommended AI packs and the Brandi Project, we have captured the upstage culture and working method in Star View.

Brandy's
Created a customer success story
RecSys & AI Challenges team
extraordinary synergy

please introduce yourself.

wonseong

hello. I am Lee Won-seong, an AI research engineer for the RecSys team (Recommender Systems) . He is responsible for solving various problems in the recommendation system with AI or machine learning technology.

Junhyun

hello. I am Junhyun Park, an AI research engineer working in the RecSys team . We are working to solve customer problems based on data.

Hyun Woo

I am Hyunwoo Kim from the AI Challenges team . I am trying to acquire new skills or achieve better performance while participating in Kaggle competitions, and in the process I was able to acquire a Kaggle Master . Other than that, I am in charge of modeling other internal projects or PoCs.

Namjoon

hello. I am Cho Nam- jun, a software engineer working in the RecSys team . We are developing the backend and data pipeline for our recommendation system.

Nice to meet you.
What kind of work are you focusing on these days?

wonseong

Recently, additional development of the Brandi AI recommendation system is underway. We are developing a system that changes the recommendation result by reflecting it in real time when the recommended item pool changes.

Junhyun

I am focusing on building a model that can universally extend the recommendation system to multiple services .

Hyun Woo

While participating in the Kaggle H&M Challenge and the ACM RecSys Challenge 2022 this year, I made many efforts to improve the technology of the recommendation system. We are concentrating on incorporating these experiences into our products and contests.

Namjoon

I am doing a lot of generalizing and automating the backend system . As the scale of development and work grows, we also do a lot of monitoring.

Recommended by Brandi Upstage
Throughout the project introducing AI Pack
I'd also like to hear a story about it.

wonseong

Brandi, a fashion shopping app, has more than 6 million members in Korea (as of September 2022), making it one of the top 3 fashion shopping apps for women. Therefore, it is a major customer for developing recommendation packs such as hyper-personalized product recommendation solutions based on various data. So, I spent a lot of time in the process of pre-tuning and preparing for various requirements. Now, starting with Brandi, we are creating a standardized process that can be introduced to other clients as well. We are committed to reducing time and providing standardized serving. Here, you can think of standardized serving as creating a 'framework' that satisfies the brandi's requirements and can provide a recommendation system not only for brandi, but also for various customer companies.

Project with Brandy
When did you start?

Junhyun

Going up to the beginning of the project itself (rather than the exact starting point) , the Challenges team and the recommended team participated in the Kaggle competition using H&M data last May. Both teams performed well enough to win silver medals. Based on the recommendation model that was verified in the competition, we proceeded to fit the brandi data. So, if you go up to the root of the project, you can say that it was from the Kaggle competition .

We brought in the best models from around the world, made recommendations suitable for brandi's sales indicators, and started AB testing from about August. Within a short period of time, tests have proven our performance. It proved to be superior to other existing recommendation systems in various indicators such as sales and clicks, and naturally led to contracts.

Upstage's recommended AI Pack is
Specifically applied to some part of brandy,
What other effects did you have?
During the project, the most important
I'm also curious about what goals you've reached.

wonseong

Currently, it is being applied in the form of personalized AI recommendations on the home page of the Brandi app . After applying Upstage's recommended AI Pack, the amount of purchase conversion per total impression increased by nearly 60% compared to before starting the business. That's almost a doubling of the +32% that Brandi's Data Optimization Office set for when it launched earlier this year. Upstage's AI model alone has increased by an average of +150% compared to the previous one.

Often, clients who want to introduce a recommendation system don't know 'what to do first' or 'what to optimize'. There are many cases where you don't know what the important indicators of your service are. Brandi had a clear metric: “Maximize Multi-Touch Attribution for Purchase Conversion Value per Impression,” or Maximize Purchase Conversion Value per Impression. In the past, 'CTR', which is a click-through rate, is often optimized, but Brandi is different from the existing problem in terms of directly optimizing an indicator that is highly related to the actual business, 'sales contribution', and I think it was able to create synergy in many ways. .

Under the common goal with customers,
I heard that you collaborated like one team.
It must have been very important yet difficult.
I would love to hear your thoughts freely.

Namjoon

I think I put a lot of thought into the process of coordinating with clients about various inquiries. Depending on the internal circumstances of the customer, the priority of inquiry/sharing may be set. The ranking of urgent issues needs to be coordinated according to the situation of the customer. So, in order to facilitate communication with clients, I tried to take good care of the process of quickly catching, making inquiries, and following-up.

Hyun Woo

Sometimes issues arise during the collaboration process. I think it is important to share issues and understand each other's current status through continuous communication. At first, it was difficult to communicate with Brandi about these things, but I did not give up and went through various processes and efforts. I believe that as a result, we were able to bring about better results because we built a stronger relationship of trust through mutual sharing, discussion, and decision-making.

Junhyun

We have improved our understanding of our customers . Actual data construction and usage The data is linked to retail and is formatted to show and store the data well. There may be cases where the recommended format does not fit the actual format. There was a process of matching each other to a more optimized format, and we also requested what we needed. So, I wonder if there was a good collaboration that could achieve more achievements.

wonseong

yes i agree too Incidentally, while most recommendation problems solve click optimization problems, Brandi was related to sales. There are relatively fewer studies on 'sales' than on 'clicks'. So, in the beginning of the PoC, I think I put my heart into designing Brandi's problem while conducting research and study. For example, 'corporate sales' can only be measured when sales actually occur. However, it is not easy to measure with offline data, so it was difficult to design such an experiment, but I am proud to be able to produce meaningful results.

After Collaborating with Brandy
Can you feel the reaction of the market?

Junhyun

Inquiries from customers in e-commerce, especially in the fashion sector, are increasing. There is also a lot of discussion about actual service progress.

Looking forward to the good news again. from now on
Standardized service even if the number of customers increases
To provide solutions to multiple customers through
I wish it would be much easier.

Junhyun

Yeah! Since we have a good model, we have created a system to set the right options for our customers . You can now tune your model to match the performance metrics other customers want!

Brandy Collaboration as an Opportunity, Inside the Recommendation Team
Or the recommendation team and the challenge team together
Points for growing up or learning lessons
If so, I'd like to hear from you.

Hyun Woo

The Brandi project was not the first collaboration with the RecSys team. While experiencing H&M Kaggle together, I studied the latest technologies as well as how to solve recommendation problems and improve performance when approached/analyzed.

We learned a lot while sharing the latest technologies and how to use them through ongoing collaboration processes, such as the 'RecSys 2022 AI Recommendation Contest' hosted by ACM RecSys, the most prestigious society in the field of recommendation systems, and the recommended AI solution for Glorang's online kids school 'Gguge' . seams like. It was also a great opportunity to experience how to naturally share/discuss within the team and continue the collaboration process, and it was very memorable.

Junhyun

Collaborating with the AI Challenges team, which has been recording good performance/scores on Kaggle, has allowed me to gain a lot of expertise and experience in the field. Through collaboration with clients, I think I was able to solve problems or requirements that had not been resolved for a long time, and learned a lot in the process. No matter how well we hold a knife, we shouldn't know how to wield it. I think the fact that we learned that was an opportunity for us to grow even further. Also, based on these learnings, it was good that the model could be further improved and Brandi's performance could be developed in a way that gradually went up.

wonseong

Since the AI Challenges team is a group of experts who improve performance, I think I learned a lot from their experiences through collaboration. We thought about how to design the problem and how to provide it to the actual customer, and the AI Challenges team thought deeply about the performance aspect, so it seems that there was a lot of synergy with each other.

In addition to the AI Challenges team, I think it was a good collaboration with various teams such as the AI Biz team and the marketing team. The Biz Team successfully completed the brandi contract, and the Communication and Marketing Teams promoted it wonderfully.

Namjoon

Collaboration with backend and researchers is the most memorable for me. When we say we are making a product (we are making an AI product), the model has no choice but to be related to the backend. I felt that the collaboration method, which continues to organically exchange as a one-team team, was a good fit, rather than a structure where the backend and models were divided. And the data needed for the model is always there. Discussions on how to deliver data continue to be needed, but I think the current organizational structure has been helpful in many ways.

In the field of recommendation AI system in the future
Do you have a goal or direction you want to challenge?

wonseong

Currently, Upstage's Recommendation Pack is focused on a rather narrow field of application called 'recommendation system', but you can think more broadly from the perspective of the whole company. We are considering expanding into a tool that helps customers' profitability optimization (profit maximization) by receiving customer data and optimizing customer KPIs, such as sales and stay time.

The journey of making difficult products together,
One team & one step more
even tighter!

What is important to you at work
'Upstage way'
I'd like to ask you a question.

wonseong

I would like to say that One team is important and a value I like. Because Upstage has a horizontal corporate culture, even juniors can work as a team with ownership, communicate horizontally with leaders, and in the meantime, creativity seems to have the effect of being maximized. Leaders have their own concerns, juniors have their own concerns, and in the middle, they may have their own concerns, but I think these create complementary synergies . While working in a horizontal organizational culture, I also learn a lot from the juniors, and of course I have a lot to learn from the leaders, and I think that's what makes Upstage more special.

Namjoon

I also like One Team the most. Our team has backend and researchers in one team. I think that these two jobs cannot be separated as we make AI products together. Because we are tied up with data, we collaborate a lot. We are in charge of the same project, but the method of “You are the backend and you are the researcher, so let’s work separately” is not suitable .

Another is that the product we are trying to make is not an easy product. Because the goal is to create something that did not exist in the world. That's why I think that sometimes the goal may be shaken. You may have doubts about whether I am doing well, but at that time, the most helpful thing is that our team is the original team, and I think it is very important to continue exchanging trust within it. I also gain a lot of strength there, and I gain a lot of faith that 'we can really achieve this goal'. So I think the original team is very important.

Junhyun

I think 'One step more' is an important value. Thinking one more time, trying one more time, thinking about how to make something easier, better, and more superior. “Trying one more time” means to us as well as to the customers who receive our services. I think it will be a very attractive point for . I can tell you that we are always striving to go one step further.

Then, in the practice of 'Upstage Way'
Do you have any concerns or difficult items?

wonseong

For me, 'One step more' is actually the most difficult. Because there is a given amount of time, but finding a better solution within that time actually takes time. I think it's always difficult how efficiently we find 'one step more' within the given time. So, 'One step more' is a very big challenge, and I think we are still in the process of finding the answer.

Namjoon

The process of organizing well for sharing after the task is important. There are many tasks with set deadlines, and once you work on the schedule and new tasks come up, you can't organize enough. There are kings of organization on the team, so they complement a lot, but I think I need to supplement in terms of sharing myself.

Hyun Woo

I think ‘sharing’ is the most important thing, but I also think it’s difficult. This is because each member of our team tends to do slightly different testing tasks. I am working on a recommendation system competition, but there are also people collaborating with the Document AI team. In situations where I have to share experiences of working on different things or have meetings, I have to convey and share what I am doing, what concerns I have, what tests I am currently solving, etc. in weekly meetings, but it is not easy.

To practice the Upstage Way well
Do you have 'my own know-how' that you try?

Hyun Woo

The AI Challenges team is thinking a lot about sharing internally. 'Let's work on the document' through another team's benchmark was one plan. I am actually benchmarking the Notion Page of the RecSys team. You did a good job organizing things, including Junhyun-nim. Looking at how others do it, I think that I am learning a little bit of know-how, and I think that the practice of 'sharing' has been very helpful while collaborating with Brandy with the RecSys team.

Junhyun

I would like to say 'One step more'. What makes our company different is that we can participate in and share in decision-making with leadership. Being able to receive it seems to be of great help to me.

In our company, unlike other companies, juniors can participate in or share important decisions with ownership. If you actively participate in this way, you will think that it is really a product I made and that it is evaluated by customers. Working with that kind of mindset, I think I naturally become 'one step more'.

Finally, future plans or
What would you like to say to other stars?

Hyun Woo

This year, Kaggle H&M Challenge , ACM RecSys Challenge 2022 , etc. have achieved good results. On the other hand, there were some regrets at the highest rank. (Although there was a good result for Brandi by supplementing that part well.) A recommendation contest will be held next year, so I will study the recommended field for the rest of this year, so that next year I can write a thesis with better performance and release a press release with better performance. It is our goal and plan for the future.

Junhyun

I am very interested in creating products that can help customers and do 'Making AI Beneficial'. I'm thinking about making a recommendation system that can be easily applied to actual service, or which direction to create a product that can help realize the mission. I hope that this process will serve as a driving force to increase the value of actual warriors.

wonseong

My goal is to finish the remaining PoCs well this year, and I think I need to prepare little by little for next year. I recently became a tech leader, and I feel a bit pressured, so I think I have to do well.

Namjoon

My goal is to automate the recommended AI Pack of Upstage so that I can respond well to customers in the future. Also, I want to say thank you for filling in the missing parts of my work with the stars I work with, and I hope the rest of the year ends well.

← Back to Blog