PantryWise
Pantry-Aware Multi-Objective Recipe Recommendation
Final Report
Overview
The growth of digital food platforms has produced countless recipe data. While recipe recommendation systems exist, many focus on popularity or simple collaborative filtering, neglecting pantry availability, diet, or sustainability. Recent research has shown the potential of machine learning methods for recipes.
For example, models like RecipeRec [1] and HGAT [2] use graph neural networks to model complex user-recipe-ingredient relations, leveraging both node/relation-level attention to improve recipe recommendation. Gao et al. [3] developed FGCN, which uses ingredient-level connections to enhance embeddings. Furthermore, multimodal approaches like MHGRR [4] combine images and text, demonstrating that photos can enrich recommendations.
However, these systems often overlook redundant ingredients (e.g., salt, oil), dietary filters, nutritional alignment, and ecological sustainability. Sustainability is central to dietary guidance; Sinclair et al. [5] emphasize aligning food choices with environmental priorities.
Problem & Motivation
Problem: Current recipe recommendations rarely incorporate sustainability, nutrition, and pantry constraints. They typically optimize for similarity or popularity, ignoring ingredient waste or alignment with users’ health needs.
Motivation: The FAO estimates approximately one-third of food produced globally is wasted annually, representing a significant environmental and economic challenge [5]. Consumers increasingly demand diet-sensitive and health-conscious solutions. By integrating pantry optimization, sustainability metrics, and nutrition filters, our system distinguishes itself from existing recommenders that rely solely on similarity or collaborative filtering.
Data Sources
We used the Food Ingredients and Recipes Dataset with Images, Food.com Recipes Dataset, and Open Food Facts Dataset. The first dataset contains a CSV file with 13,582 rows with each row containing the columns of index, ingredients, recipe name, image name. There is also a folder with the respective images corresponding with the image name column. The second dataset featured over 230,000 recipes with ingredients lists and approximate nutrition ~1.1M user ratings. The third dataset is a CSV file containing around 2 million products with their respective Nutri-score and Eco-score.
Data & Preprocessing
-
Dataset 1: Foods Ingredients and Recipes Dataset with Images
a. Ingredient Normalization:
We normalized each ingredient string into a “name-only” form. This includes stripping parentheses and unicode fractions, lowercasing text, removing numerical quantities and ranges, measurement units (cups, ml), quantity phrases (to taste, plus more), and some descriptors (chopped, thinly sliced).
b. Quantity and Unit Parsing:
We preserved the ingredient quantities by parsing the ingredients into the form {‘quantity’: float, ‘unit’: str, ‘name’: str} and store them into a parsed ingredients column. The preservation process entails integer, decimal, unicode fraction handling; regex to separate leading quantity and candidate units; unit validation against a curated units list; and production of reliable fields for future quantity-based scoring even when the unit is absent. With the “1 (3½–4-lb.) whole chicken” example, the parsed ingredient shows: {‘quantity’: 1.0, ‘unit’: ‘’, ‘name’: ‘whole chicken’}.
c. Ingredient Lemmatization and Fuzzy Matching:
To find accurate pantry coverage we also implemented ingredient lemmatization and fuzzy matching. Each ingredient is further normalized by removing common descriptors and lemmatizing words to root forms (apples becomes apple). We also used the rapidfuzz library’s token_set_ratio to perform fuzzy matching between normalized ingredients and pantry items. This allows partial matching between words like “salted butter” and “butter” as in the real world, this would be an appropriate substitution. This yields an increased and more flexible pantry coverage calculation. On a similar note, aliases for ingredients were used to avoid referring to essentially the same ingredient using multiple names. An example is aliasing “scallions” with “green onions” and “granulated sugar” with “sugar”.
d. Feature Extraction:
For each recipe, we convert the normalized ingredients into a space-joined string (bag-of-ingredients) and build a TF-IDF matrix using sklearn.feature_extraction.text.TfidfVectorizer with max_features = 5000, min_df = 2, and ngram_range = (1, 2). This makes a high dimension sparse representation to capture term importance and common collocations (olive oil).
e. Image Processing:
Images were preprocessed using ResNet50 input requirements which entailed resizing images to 224x224 pixels, color conversation to RGB, pixel value normalization, and PyTorch tensor conversion. Additionally missing or corrupted images were replaced with zero-valued embeddings for a total of 13,501 non-zero-valued embeddings. -
Dataset 2: Food.com
a. Parsing Ingredient and Nutrition List:
We converted string representations of Python lists in the CSV files into actual Python lists. If the parsing of the lists fails, then we return an empty list. And then if the recipe has an empty list, it is removed.
b. Ingredient Normalization:
We normalized the ingredients by lowercasing, trimming whitespace, removing quantities and units (like “2 cups”, “1 tsp”, “3 oz”, “150g”), removing descriptive words (like “fresh”, “diced”, “frozen”, “optional”), and cleaning punctuations. The final ingredient list was joined into a single comma-separated string.
c. Nutrition Extraction and Normalization:
The original nutrition list is expanded into calories, fat, sugar, sodium, protein, sat_fat, and carbs. The original values for nutrition were per serving, and we converted it into nutrition values for per 100g. -
Dataset 3: OpenFoodFacts
a. Data Cleaning:
There were a lot of non-valid Nutri-scores and Eco-scores (na values). After filtering out the na values, there were 200,000 valid nutri-scores and 150,000 valid eco-scores.
Modeling Approaches
- K-Means
Here K-Means is combined with features like TF-IDF cosine similarity, pantry coverage, and sustainability score into one score based on weights. We use K-means on TF-IDF features to cluster into 20 recipes. The recommendation system combines the metrics mentioned above to produce a list of top-N recipes for the user.
The first metric is TF-IDF similarity, utilizing cosine similarity between the user’s pantry (bag-of-ingredients) and each recipe’s TF-IDF vector. The 2nd metric is cluster affinity which compares the pantry to cluster centroids. Next, we look at pantry coverage which looks at the percent of the normalized ingredients in the candidate recipes. This supports the main ethos of sustainability, as users can just use ingredients they already have. Then we have a complexity penalty. For example, recipes with ingredient lists of 3 or less get downranked as those are probably just things like sauces which are not desired for cooking. Finally, we look at sustainability where total recipe emissions (kg CO2e) become transformed with an inverted equation: score = 1 / (1 + emissions) such that lower emission recipes get higher scores.
The final score is a weighted sum of TF‑IDF similarity, pantry coverage, cluster score, and the sustainability score with default weights of 0.3, 0.3, 0.2, 0.2, respectively. This model preserves relevance while also accounting for practicality and environmental impact. The top-k results are returned with their respective scores. - LightGBM
The pantry-aware recipe recommender is a system designed to suggest recipes users can cook with the ingredients they already have, while balancing taste, nutrition, and sustainability.
It is built on three LightGBM models: a taste classifier trained on Food.com ratings (predicting whether a recipe is “GOOD”, meaning having a rating of 4.5 of greater), and two regressors trained on Open Food Facts to estimate Nutri-Score (healthiness) and Eco-Score (environmental impact).
Additionally, feature engineering combines nutritional macros, recipe metadata, and compact text embeddings derived from TF-IDF + SVD. Taste is modeled as a binary decision because regression on ratings proved too noisy, whereas classification preserves the key signal: whether a recipe is good enough to recommend.
At recommendation time, the system filters recipes by pantry coverage (≥80%) and computes a weighted final score that combines predicted taste, nutrition, and sustainability. The model then ranks and returns the top recipes. - CNN
This model recommends recipes by analyzing food images and computing visual similarities between them using convolutional neural networks. Rich features of food images are extracted for content-based recommendations without relying on text.
The model is built on a pretrained ResNet50 architecture, extracting 2048-dimensional image embeddings. Standard preprocessing is done including resizing to 224x224 pixels, RGB conversion, and normalization. The embeddings then carry information about the food images such as appearance, texture, color, and presentation.
At recommendation time, cosine similarity is computed between the image embeddings and the queried image. The top-N most visually similar images are returned after being ranked. This allows dish recommendation to be based on recipes that look similar to the user’s queried image, promoting recipes that can be created using similar ingredients as the user.
The model also includes quantitative evaluation metrics including hit rate@K, mean reciprocal rank@k, and precision@k. These metrics quantify if relevant results appear in the top k rankings, ranking quality, and recommendation accuracy, respectively.
Results & Evaluation
Model 1 · K-Means Pantry Ranking
Results
When running the algorithm, the top k recipes are returned with their respective scores:
This visual illustrates the 20 distinct clusters that emerged which represent the different types of cuisines.
The PCA also shows reasonable separation between the clusters. For some clusters there tends to be heavy overlap, but that is mostly as a result of certain ingredients being featured in many cuisines rather than a result of the model.
Model Evaluation
Silhouette Score: We obtained a moderately positive silhouette score, 0.0106, with k = 20, indicating that clusters are reasonably cohesive but not strongly separated, which is similar to the result of the PCA.
Davies-Bouldin Index: Our DB index, 7.0392, (for k = 20) reflects the same pattern as the silhouette score; clusters are distinct but do show some overlap due to ubiquitous ingredients (oil, salt, onion, etc.).
From the elbow analysis, it appears that our choice of k = 20 is in the optimal range of k as the WCSS continuously decreases after k = 20.
These metrics align with the recommendation model’s behavior. For instance, two porterhouse steak recipes fall into separate clusters (17 and 19) due to minor variations in preparation, yet both achieve high similarity and pantry coverage. This demonstrates that clusters provide useful neighborhoods for narrowing the search space while the final ranking ensures relevance.
Model Performance
For the average pantry coverage, the baseline model covered 68% while the K-means model covered 72%. This increase in coverage helps ensure that core ingredients are favored over garnishes like herbs and spices.
The recommendation diversity of the K-means model was a lot higher than that of the baseline model. The baseline model often returns recipes with nearly identical ingredient lists.
The sustainability metric of the baseline model was quite simplistic as positive equated to “good/sustainable” while negative equated to “bad”. The K-means model improves and quantities the impact of an ingredient by using its carbon footprint. For example, the K-means model utilizes the weight of the ingredients to generate carbon emissions estimates for each ingredient. This enables a more context-specific estimate of sustainability score and allows us to state that Recipe X has a 33% lower carbon footprint than Recipe Y.
We realized that there would be an inverse relationship between the recipes with high pantry coverage and the complexity of the recipe. Simpler recipes with minimal ingredients would tend to have high pantry coverage. Therefore, the complexity of the recipe needed to be accounted for.
Ingredients were normalized and lemmatized to improve ingredient matching/aliasing and reduce ingredients to a “name-only” form, avoiding extraneous phrases like “to taste” and “plus more”.
The final score was a weighted sum of TF‑IDF similarity, pantry coverage, cluster score, and the sustainability score with default weights of 0.3, 0.3, 0.2, 0.2, respectively. This model creates practical recipe recommendations that balances the recipes one can make (pantry coverage), the recipes that suit already existing ingredients (similarity/clustering), recipes worth making (complexity), and recipes that are environmentally friendly (sustainability score).
Limitations
The quantity of ingredients was a bit difficult to extract. For example, our normalizer struggled with complex quantities such as “1 (14.5 oz) can of diced tomatoes” or “2 8-oz packages of cream cheese”. The failure came from misattributing the descriptive text as part of the unit or quantity. This impacted the sustainability score because if an ingredient was not parsed property, then the carbon footprint calculation would also become inaccurate.
Additionally, the weight of ingredients was estimated based on common whole-item approximations when no explicit unit was provided in the recipe. This led to carbon footprint estimations not being fully accurate for such ingredients with missing weights.
Some of the ingredients did not have carbon footprint data. Instead of using that they had a carbon footprint of 0, we used a default nonzero value of 5.0. This means that the sustainability score of the recipe was a rough estimate and not completely accurate.
The list of ingredient aliases can be improved to have a wider coverage of all the terms that are used to refer to identical or similar ingredients.
Model 2 · LightGBM Multi-Objective Ranking
Results
The taste model is the filter for the recipes that taste good (or have a rating of above 4.5). The nutrition model predicts the nutrition values of the recipes for by grading each ingredient and categorizing them from A to E. The sustainability model is good at predicting and ordering products by sustainability.
Using the 3 models together, the top 10 recipes are returned based on the weight of the output from each model with the final score being determined by 0.4 × Taste + 0.3 × Nutrition + 0.3 × Sustainability. This model slightly prioritizes taste to align with user preferences because even if a recipe is healthy, if it does not taste good then the user will not eat it.
Model Evaluation
From the table on the bottom right, the recall of GOOD recipes is 0.962, which means that there is minimal loss of highly rated dishes. When the model predicts GOOD, it only correctly predicts so 58.9% of the time. The other 41.1% of the recipes predicted as GOOD have a rating below 4.5. The false positives that the model is predicting are recipes typically with a rating of 4.0-4.5, which is still a relatively good rating.
For the normalized confusion matrix, it can be seen that the model leans heavily towards predicting GOOD. This model is optimized so that good recipes are caught even if it means letting the recipes with a rating close to the threshold be classified are NOT_RECOMMENDED. This creates a large amount of recipes that will later be filtered out by nutrition and sustainability scoring.
Having an R² score of 0.923 means that the model explains 92.3% of variability in Nutri-scores. This score is very close to 1, meaning that our model was a good fit for our dataset. The RMSE of 2.46 is small relative to the range of nutri-score (-15 to 40). This error is smaller than the difference between the nutri-score categories (which are around 10 to 15 points apart).
For the sustainability model, it had a higher RMSE of 13.29. However, the eco-score ranged from 0 to 100, which is 5 times larger than that of the nutri-score. The fact that the RMSE score is larger makes sense that the RMSE of this model is higher than that of the Nutritional model. The larger RMSE also accounts for the fact that sustainability is harder to predict than nutrition due to more complex factors such as country of origin, packaging, transportation, and agricultural practices. The R² score of 0.695 illustrates that this model has a relatively good prediction of environmental impact. The Spearman’s rank correlation coefficient of 0.836 shows that this model correctly orders products by sustainability 83.6% of the time, which is extremely helpful for recommendation systems because this model reliability tells us which product is more sustainable than another.
The precision @ K = 5 is higher than that of the precision @ K = 10. This is acceptable for a recommendation system as a smaller K results in a more niche and narrow set of items. As the K increases, items that are less relevant are added to the consideration set, which decreases the precision. The nDCG@K for both K values are above 99%, which means that the quality of our ranking is extremely good, regardless of the K we choose.
Compared to the baseline model, our model saw improves in all three aspects. The nutrition score improved by 14.53, the largest of all 3 models. This is because it is the most straightforward to predict (this is gathered from the very high R² score). The sustainability model also improved by a fair amount as a result of its strong ranking quality (Spearman ρ = 0.836). The taste only improved by 0.32. However, this is because the baseline model was already quite high (4.46/5.00) and the ratings within the dataset were skewed towards recipes rated between 4.0 and 5.0.
Model Performance
This model shows strong performance for nutrition prediction, good ranking consistency for sustainability, and high recall for good-tasting recipes. Compared to random pantry-matched recipes, the recommender recommends dishes that are healthier, more environmentally friendly, and slightly better-rated, demonstrating meaningful multi-objective improvements while keeping recommendations practical and user-relevant.
The strengths of this model were high recall for good recipes (96%), ensuring minimal loss of highly-rated dishes. This model also had strong nutrition prediction (R² = 0.923, RMSE = 2.46) and good sustainability ranking (Spearman ρ = 0.836). This was an effective multi-objective ranking because it improves both nutrition and sustainability scores over random selection while maintaining taste.
Limitations
Text Domain Mismatch: The sustainability model is trained on OFF product_name (e.g., “Organic Almond Milk”) but applied to Food.com ingredient_text (e.g., “almond milk, honey, oats”). Despite this mismatch, the R² of 0.695 and high ranking correlations (Spearman ρ = 0.836) suggest the model generalizes reasonably well, likely due to: shared food vocabulary across domains, macro nutrients as primary features (text as secondary), and semantic similarity in TF-IDF space.
Taste Classification Accuracy: The 62.5% accuracy is only slightly above the 62.2% majority baseline. However, the 96% recall for GOOD recipes is what matters for our use Case. We prioritize not missing good recipes over perfect precision.
Ingredient Filtering: The greater than 5 ingredients filter is a heuristic to exclude simple recipes. This threshold may need adjustment based on user feedback.
Fine-tune Taste Threshold: Instead of using 4.5 as the cutoff, we can experiment with other cutoffs such as 4.3 or 4.4 to see how the precision and accuracy is affected. The original threshold was set at 4.5 because so many recipes in the dataset were very highly rated. Playing around with the cutoff could also decrease the amount of false positives being predicted by this system
Model 3 · ResNet50 Visual Retrieval
Results
For a queried recipe, when the convolutional neural network model is run, the recipe’s top k closest recipes are returned using the ResNet50’s 2048-dimensional embeddings.
The hit rate, mean reciprocal rank, and precision results show that the system is effective in visual retrieval for k = 10 for example, with HR@10 = 0.87, MRR@100 = 0.49, and P@10 = 0.30 for example.
Model Evaluation
At k = 10, the model reaches 91% hit rate, meaning that users find at least one relevant recipe in their top ten at least 91% of the time. At k = 20, this increases to 94%. Then the MRR shows us the first relevant recipe appears earlier in the ranking. With MRR@10 = 0.490, this means that the average position of the first relevant recipe is (1/0.490) around second. This shows good ranking quality from our system. Precision is relatively stable (0.29-0.32) even while changing k, showing us that consistent relevance persists throughout the list instead of just the top few items. Thus the metrics show us that k = 10 is a good tradeoff for us as it has good recall while maintaining good ranking quality and precision.
In addition to these metrics, we also look at average neighborhood stability under noise perturbation, resulting in 92%, showing robustness in the model. This tells us that the embedding space is smooth and continuous, meaning reliable results despite variations in input images. However the embeddings also show 54.5% zero values and low L2 norms (12.1 mean) showing normalization issues. This sparsity may reduce the embeddings’ distinctiveness while the normalization issue might affect similarity scores.
The distribution also shows small inter-image similarity (0.338 mean) with a reasonable standard deviation (0.132). The 37.6% reciprocal neighbor ratio also suggests some asymmetry. When A considers B a close neighbor, the contrary is true only about a third of the time. Despite these limitations, the model achieves strong performance as shown with the hit rate and mean reciprocal ranking.
Model Performance
From the strong hit rate and mean reciprocal ranking at k = 10 of 91% and 0.490 respectively, strong practicality of the model is shown. The precision stabilizes around 31% across different list lengths. Additionally the system has persisting relevance from 29% at k = 1 to 94% at k = 20. With 92% neighborhood stability with noise perturbations, the embeddings show robustness under pressure which is reliable for real conditions. So the model captures visual similarity relationships for recipe discovery allowing users to find alternates to their queried dish even across other food groups.
Limitations
Despite the strong result with the hit rate and mean reciprocal rank, there are some limitations. The sparse embeddings with 54.5% zero values might artificially suppress similarity scores. The moderate reciprocal neighbor ratio of 37.6% also suggests asymmetry where image similarity from embeddings is not necessarily bidirectional in some cases. The model also cannot infer nutritional content or ingredient quantities. Finally 18.4% of recipes had missing images replaced with zero embeddings in extraction, potentially reducing recommendation quality. These factors show the need for multimodal approaches to address ingredient-level and sustainability-aware matching.
Model Comparison
The three models each address recipe recommendations through different methods, each offering their own strengths and limitations. The K-means clustering model’s greatest strength is offering ingredient-level optimization through the TF-IDF similarity between the recipe ingredients and pantry as well as sustainability scoring on carbon emissions. It has practical sustainability strengths by explicitly minimizing food waste with pantry utilization greater than 80%. However the performance is still limited as it has ingredient parsing challenges alongside just estimating sustainability, relying on approximations and weight assumptions.
The LightGBM-based model provides the most accurate nutritional and sustainability prediction metrics through the train regression models. With R² scores of 0.923 for nutrition prediction and Spearman ρ = 0.836 for sustainability, it achieves the best performance in these domains while having high recall for well-rated recipes. The limitations, however, start with domain mismatch between training data sources and the moderate taste classification accuracy of 62.5%, which prioritizes recall over precision.
The CNN model offers the most intuitive recommendations through image embedding comparisons. It achieves a 91% hit rate at k = 10 with good robustness as its neighborhood stability is 92%. It addresses visual and presentation based discovery, which is more similar to mainstream models, something that the other models neglect. However, this model provides the least explicit sustainability or nutritional output and suffers from embedding sparsity that may limit discrimination.
Overall, the K-means model is best for pantry optimization, LightGBM is good for quantitative health and sustainability metrics, while CNN best shows visual rankings. For a comprehensive recommendation, a hybrid system could utilize the convolutional neural network ranking, K-means pantry optimization, and LightGBM’s sustainability predictions to balance visual appeal, practicality, and impact. This multimodal approach would address the various core challenges of satisfying preferences that also are sustainable. This approach separates our work from conventional popularity or rating based recommenders.
Future Work
For model 1, we could improve ingredient parsing, include water usage and land use for sustainability, optimize clustering by experimenting with Hierarchical DBSCAN, and include advanced features such as seasonal awareness, budget optimization, meal planning, dietary restrictions.
For model 2, to improve the model, we can fine-tune text features by training a domain-specific embedding model on food text, add user personalization by incorporating dietary restrictions, allergies, taste preferences, improve sustainability predictions by using actual ingredient-level LCA data, optimize hyperparameters by using grid search, and using A/B testing to validate multi-objective weight with real users.
For model 3, future work will be on enhancing recommendation accuracy, embedding quality, and sustainability. We can augment the current system with multimodality, fusing CNN-based similarity with text-based ingredient TF-IDF for both visual and compositional balance. We can also improve embedding quality by applying L2 normalizing and reducing data sparsity. Finally to address sustainability goals, we can integrate with the LightGBM sustainability pipeline for explicit carbon footprint estimates, creating a true sustainability-aware system.
Additionally, as previously mentioned, for a potential future step, combining all three methods for a multimodal approach could be beneficial for balancing the tenets of our project: user preference, sustainability, and health.
References
[1] Y. Tian, C. Zhang, Z. Guo, C. Huang, R. Metoyer, and N. V. Chawla, “RecipeRec: A Heterogeneous Graph Learning Model for Recipe Recommendation,” arXiv.org, 2022. https://arxiv.org/abs/2205.14005
[2] Y. Tian, C. Zhang, R. Metoyer, and N. V. Chawla, “Recipe Recommendation With Hierarchical Graph Attention Network,” Frontiers in Big Data, vol. 4, Jan. 2022, doi: https://doi.org/10.3389/fdata.2021.778417.
[3] X. Gao, F. Feng, H. Huang, X.-L. Mao, T. Lan, and Z. Chi, “Food recommendation with graph convolutional network,” Information Sciences, vol. 584, pp. 170–183, Jan. 2022, doi: https://doi.org/10.1016/j.ins.2021.10.040.
[4] R. Ouyang, H. Huang, W. Ou, and Q. Liu, “Multimodal Recipe Recommendation with Heterogeneous Graph Neural Networks,” Electronics, vol. 13, no. 16, p. 3283, Aug. 2024, doi: https://doi.org/10.3390/electronics13163283.
[5] M. Sinclair, E. Combet, T. Davis, and E. K. Papies, “Sustainability in food-based dietary guidelines: a review of recommendations around meat and dairy consumption and their visual representation,” Annals of Medicine, vol. 57, no. 1, Mar. 2025, doi: https://doi.org/10.1080/07853890.2025.2470252.
Project Timeline
Gantt Chart (Accessible on @gatech.edu email)
Team Contributions
| Team Member | Contributions |
|---|---|
| Eileen Chen | M1: Cleaning; M3: Implementation; Report |
| Euan Ham | M1: Cleaning, preprocessing; M3: Implementation, evaluation, visualization; Webmaster |
| Taeho Kim | M1: Implementation, M2: Implementation, Evaluation; Final Presentation Materials |
| Inho Lee | M1: Implementation, M2: Implementation, Evaluation; Final Presentation Materials |
| Kimberly Tsung | M1: Evaluation; Report |