This study presents a new method to overcome the limitations of image-based food quantity estimation capabilities of multimodal large-scale language models (MLLMs). MLLM is great at recognizing a variety of foods, but struggles to accurately estimate individual food portions from photos of actual meals. This study added a small network (portion head) with enhanced geometric information to a fixed commercial MLLM, thereby dramatically improving the accuracy of food quantity estimation while maintaining the strong food recognition ability of MLLM. This method works with a single RGB image without a depth sensor and does not require separate fine-tuning of the MLLM.

1. Introduction: Importance of meal size estimation and limitations of MLLM 🍽️

Dietary assessments are essential for nutritional research and public health monitoring, but traditional methods are time-consuming and at high risk of bias. For example, the 24-hour recall method is detailed, but it places a burden on respondents and is prone to recall bias. Food frequency questionnaires are simple, but they can be difficult to obtain detailed information about specific foods and amounts.

To solve these problems, Image-based food recognition and meal size estimation technology is emerging as an alternative. This technology can provide detailed and objective information without burdening the respondent. However, existing methods had limitations in that they were developed on small, controlled datasets and were difficult to apply in real environments. In particular, there was a tendency to report only the total amount of the meal (mass, calories, etc.), making it difficult to determine the type and amount of individual foods, making it difficult to calculate important nutritional scores such as the Healthy Eating Index (HEI).

Multimodal Large-Scale Language Models (MLLMs), recently trained on web-scale image-to-text data, demonstrate outstanding ability to "zero-shot" recognize various foods even from uncontrolled real-world photos, and are already being utilized for image-based food recognition and nutrition estimation in modern nutrition apps. However, MLLM showed weakness in precise meal size estimation**, resulting in large discrepancies in energy and nutrient estimates.

To overcome these limitations of MLLM, this study proposes a method to enhance the capabilities of commercial MLLM by developing a dedicated meal size estimation "portion head". The head utilizes MLLM as a fixed parent detector. MLLM provides a name, bounding box, and density range for each food, and the head analyzes this information and the entire image collectively to estimate the mass of each food. This process does not require a depth sensor, nor does it require any fine-tuning of the MLLM itself.

1.1. Key Contributions ✨

1. Development of geometric information-enhanced head for accurate meal size estimation:
   • Image×Food Cross-attention Transformer: Comprehensively analyzes all detected foods and images to effectively divide and calculate the areas shared by foods.
   • Structured Softmax-Ownership Volume: This accurately predicts the mass of individual foods. Food-specific errors were significantly reduced compared to MLLM alone without a depth sensor or MLLM fine-tuning.
2. Building a practical system suitable for large-scale diet evaluation:
   • Extracts various foods from photos of actual meals and estimates individual food quantities, allowing linking with standard food databases to obtain precise nutritional information (macronutrients, micronutrients, diet quality scores, etc.).

2. Related research: Advances in nutrition research and computer vision 📚

Image-based food recognition and meal size estimation have received much attention in the fields of nutrition research and computer vision.

2.1. Nutrition Research & Apps 📱

Image-assisted and image-based diet assessment methods have been developed over the past decade, and more recently, AI-based methods including Convolutional Neural Networks (CNN) and MLLM have been proposed and evaluated. Although AI-based imaging has shown great promise, large and inconsistent errors remain a major barrier to widespread adoption in population-scale diet assessments. Consumer nutrition apps like MyFitnessPal and Noom have also adopted image-based food tracking, but there is still a significant quality gap from research-level dietary assessments.

2.2. Computer vision field 🧠

Recovering mass or volume from food images is a key challenge. This is because 2D photos lose the 3D information that determines the amount of food. Early monocular methods regressed quantities or energies directly from a single RGB image. To restore depth information, datasets such as Nutrition5k provided dish-specific depth maps, and DPF-Nutrition predicted depth from RGB images and used it for nutrition estimation.

More recently, methods have emerged that explicitly exploit geometric information by reconstructing 3D food models or point clouds and inferring scale from physical references or contextual objects. Although these geometry-based methods have increased accuracy, they have the limitation of relying on depth sensors, multi-view capture, or unstable monocular 3D reconstruction, and operating with a closed food vocabulary.

On the other hand, this study uses a single RGB image without explicit depth information. While MLLM provides open-vocabulary food identification, it learns structured volumes from fixed appearance features.

3. Method: Combination of MLLM and geometric information enhancement head 🛠️

Our system combines a stationary commercial MLLM with a small, trainable geometric information enhancement head. MLLM provides open lexical semantic information such as the name, bounding box, and density range for each food, and the head supplements geometric information that MLLM lacks, transforming appearance features into calibrated food-specific quantity estimates.

Figure 1 below shows the overall data flow of the system. <br> Figure 1: Geometric information-enhanced MLLM architecture. Fixed MLLM (Gemini-3.5-Flash) extracts food-specific names, bounding boxes, and density ranges from RGB images, and fixed DINOv2 ViT-S/14 provides patch tokens. The Image×Food encoder and Image-only encoder generate exclusive patch-specific ownership partitioning, food-specific density fixed to the MLLM range, and patch-specific height fields through structured heads. The volume estimate vn for food n is the sum over all patches p, and the food-specific mass estimate mn is vn ⋅ ρn.

3.1. Multimodal LLM and Vision Backbone 🏗️

MLLM is queried as a fixed API without any fine-tuning. We adopted Gemini-3.5-Flash by benchmarking five leading multimodal models. This model had the best direct meal size accuracy of any benchmarked model along with strong recognition capabilities. For each N detected foods, MLLM returns a name, a bounding box enclosing all pieces of that food, and a density range [ρmin, ρmax]. The names are embedded into the CLIP text encoder and act as the main semantic signals, while the bounding boxes and density bounds serve as geometric and physical dictionary information.

The vision backbone is a fixed DINOv2 ViT-S/14. We fix the size of all images to 336x336 pixels, resulting in a grid of 24x24=576 patch tokens. These patch tokens are cached to improve speed. Neither MLLM nor the vision backbone are updated.

3.2. Image×Food Encoder 🔍

This encoder generates a contextualized food embedding for every food detected, providing evidence of ownership and density used by the head, along with per-patch memory. Each layer is a transformer decoder style block, consisting of self-attention for food tokens, cross-attention for patch memories, and a feedforward sublayer. Since foods form an unordered set, we do not use causal masking.

<br> Figure 2: Image×Food encoder block. L=4 Each layer is a transformer decoder style block. There is multihead self-attention to food query tokens, multihead cross-attention from food to patch memory memn,p that recognizes food-specific geometric information, and a feedforward sublayer. Each sublayer is wrapped with residual connections and layer normalization. The stack outputs a contextualized food-specific token Zn. The density head reads only Zn, and the ownership head scores the (memn,p, Zn) pairs to produce the ownership logit.

• Input:

  • N food query tokens: generated by projecting each food's bounding box features, density features, and CLIP name embeddings onto the model width d.
  • Memory recognizing food-specific geometric information: For a food n and a patch p, the memn,p entries are projected onto the d width by concatenating the fixed DINOv2 patch token xp and the geometric features that describe where the patch p is located relative to the bounding box of food n.

• Layer structure: Queries are passed through four stacked layers. Each layer applies three sublayers:

  • Food↔Food Self-attention: Allows food tokens to coordinate with each other to divide shared areas without double counting.
  • Food→Patch Cross-attention: Food token attends to its memory slice memn,·. Geometric information is placed in attention values ​​so that the model can read them as content and make inferences.
  • Position-wise Feed-forward Network: Applies to each food token.

• Output: Contextualized food-specific token Zn. The density head reads Zn directly, and the ownership head scores the (memn,p, Zn) pairs to produce the ownership logit.

3.3. Image-only encoder 📏

This encoder predicts one positive scalar per patch, the "pile-height" field 0ptp, based solely on appearance information, regardless of what food is present in the image.

• Structure: DINOv2 patch tokens are linearly projected to d widths and then passed through a shallow patch self-attention encoder. This allows each patch to leverage the spatial information around it (e.g., a tall middle patch vs. a thin edge patch). The patch-specific MLP then maps each token to a scalar and uses the softplus activation function to output the height such that 0ptp > 0.
• Input/Output: The input is only a fixed patch token, the output is a field {0ptp}* for a 24x24 patch grid. This height field does not depend on the food set, so it is a reusable estimate of how much is raised from the surface of the plate, shared by all foods when integrating volumes.
• 0ptp is the potential height, not the predicted depth.
  • 0ptp serves as a depth map in depth-based meal estimation methods, but is not intentionally intended to predict depth. There is no depth supervision or loss function during training, 0ptp is learned end-to-end with downstream mass objectives. This serves to make 0ptp match the measured mass when the integrated volume is multiplied by the density. Therefore, this is a unitless potential patch-specific weight rather than a calibrated measurement surface, and is defined at a coarse patch resolution of 24x24 rather than at the coarse pixel level. This is a design choice, because while explicit single-view depth estimation (sensor depth, pseudo-depth, depth-supervised height heads, etc.) showed negative results in our setup, the latent height learned under mass supervision can recover the missing vertical dimension while keeping the model depth-free.

3.4. Structured prediction head 📊

This head takes three inputs from the encoder:

• Contextualized food-specific tokens Zn and food-specific patch-specific memories memn,p obtained from the Image×Food encoder.

• Height per patch obtained from Image-only encoder 0ptp. Use this information to estimate the volume, density, and mass of each food.

• Ownership: For each patch, answer "Which part of this patch belongs to each food?"

  • Generate a claim logit zn,p for each food-patch pair and a background logit zbg,p for each patch. zn,p indicates how strongly food n claims patch p, and zbg,p indicates whether patch p is a plate or an empty space.
  • Apply softmaxes to N foods and backgrounds to compete for each patch. As a result, on,p ∈ [0,1] becomes the proportion at which patch p is assigned to food n.
  • This exclusive, soft partition makes the downstream volume work well. Since the area of ​​a patch cannot be counted twice, overlapping foods must split a shared area. Larger foods take up more of the patch, and the background absorbs the plate area without inflating the volume of any food.

• Density: For each food token Zn, the food-specific log density is predicted, fixed to the range [ρmin, ρmax] of the MLLM. This ensures that ρn remains within a physically plausible range while also ensuring that the model is calibrated within that range.

• Volume and mass: The volume vn of food n is calculated by integrating the shared height fields of the patches it possesses, and the mass mn is inferred from the density ρn:

  • vn = Σp on,p 0ptp
  • mn = vn ⋅ ρn
  • Total amount of dishes Mdish = Σn mn
  • Plate total is simply the sum of the masses of each food item. We intentionally omitted the learned dish sum residual head. This helps with dish total accuracy, but can overfit food-specific goals.

3.5. Training 🏋️‍♀️

The query is a MLLM detection result and no ground-truth food list is provided. Ground truth mass by food is the only supervisory goal.

• Match-and-mask supervision:

  • Detection results do not have to match ground-truth ingredients, so we align detection results one-to-one with ground-truth foods. Nutrition5k uses fuzzy name matching, FPB with ground truth bounding boxes, and NV-Real uses token and descriptor based name matching along with Intersection over Union (IoU).
  • This results in a match-and-mask approach. This is similar to matching-based supervision of aggregate prediction models such as DETR, but unmatched queries are masked in the loss calculation and are not pushed towards the "no objects" goal.
  • Specifically, food-specific mass loss applies only to matched detections, and the density hinge locks all detections (regardless of whether they match or not) at their valid densities. Unmatched detections remain as lossless queries, and appearance-based masses are summed during inference.
  • The backbone remains fixed and is optimized with the AdamW optimizer. The structured volumetric model is warm-started from a simpler direct scalar-volumetric model (equivalent to structured ownership removal ablation). This is because training structured volumes from scratch is less stable.

• Loss: The food-specific mass target is the sum of two Smooth-L1 terms and a density hinge: *L = Σn∈matched [SmoothL1(log mn, log gn) + α SmoothL1(mn, gn)] + λρ Σn hinge(ρn, [ρmin,n, ρmax,n])

  • where gn is the ground truth mass, mn is the predicted mass, α is the linear-gram weight, and λρ is the density-hinge weight.
  • The log space term is scale invariant. That is, it penalizes relative errors, so a few grams of garnish and a large amount of rice will contribute similar slopes despite double-digit mass differences, keeping the optimization stable over a wide mass range. But scale invariance also means there is less penalty for absolute errors in large foods.
  • The linear-gram term adds a penalty measured directly in grams, putting pressure on underpredictions for large amounts of food that would be tolerated by the logarithmic term alone. α is kept small to provide enough linear signal to correct for large errors, but ensure that the noisy heavy-portion tail dominates and does not harm small and medium-sized foods.

4. Experiment: Performance verification on real datasets 🧪

We show that adding a lightweight geometry head to a fixed commercial MLLM significantly improves food-specific meal size accuracy over gram estimates from the MLLM itself. We focus on the food-specific accuracy, i.e. the identified mass, required for diet assessment.

The evaluation was conducted on three real-world datasets: Nutrition5k, Food Portion Benchmark (FPB), and NutritionVerse-Real (NV-Real).

4.1. Dataset 📊

1. Nutrition5k: A controlled benchmark consisting of approximately 5,000 restaurant meals. The mass of each ingredient was measured continuously, and approximately 3,500 foods were captured overhead with a stationary RGB-D instrument. We use this subset to evaluate it on a holdout test set of 502 foods.
2. FPB (Food Portion Benchmark): A large dataset containing 14,083 photos across 138 food classes from Central Asian cuisine. Each dish was photographed from multiple viewpoints (a top view and four side angles). We trained and tested on this pool of mixed-angle images to test generalization ability to different capture geometries and unfamiliar dishes rather than a single fixed viewpoint. Evaluated on test splits of 2,197 images.
3. NutritionVerse-Real (NV-Real): A dataset consisting of 889 real consumer smartphone images. Spanning 251 dishes, every ingredient was weighed individually. Images were taken from free angles without a fixed camera-to-food distance, thus reflecting the realistic, uncontrolled smartphone capture settings that deployed diet tools must deal with. Evaluated on test splits of 265 images.

Protocol: For each dataset, heads with the same architecture, loss function, and hyperparameters were trained on the training split of that dataset and evaluated on the holdout test split. In all datasets, recognition is completely open-vocabulary. MLLM names foods freely and never looks at class menus or food-specific lists in training or testing.

4.2. Key results: Meal size estimates compared to MLLM 📈

We compared food-specific meal size accuracy using food-specific Mean Absolute Error (MAE) and Percentage MAE (PMAE). For statistical stability, models were run three times and averages are reported.

Table 1: MAE/PMAE by food: our model vs. MLLM only. (lower is better)

The results show that Geometry Head reduced the meal size error for each food item by 33-41% compared to MLLM alone. This improvement was consistent across both controlled (Nutrition5k, -41%) and uncontrolled real data (FPB, -35%; NV-Real, -33%). 😮

4.3. Estimated amount of food compared to the original paper 🏆

Since the original papers in the three datasets report different fine-grained accuracies, we compared them with our model on the metrics reported by each paper. All comparisons were made in an image-only setting without depth information.

Table 2: Estimated amount of food compared to the original paper (image only, no depth). (The lower the better.)

Our model outperformed the dish-total MAE reported by the original papers in Nutrition5k and NV-Real, and also performed better on food-specific MAE in FPB. In particular, the Nutrition5k paper reported that the depth/volume model achieved 29.4 g, but this was due to using depth information. We achieved even lower errors than this without any depth information!

For reference, the Nutrition5k paper reported human meal size estimation error for a small number of foods, with an average per-image percentage error of 53% for non-experts and 41% for trained nutritionists. 🧐

4.4. Ablation study ✂️

We measured food-specific MAE on the Nutrition5k dataset, removing each component one by one from the full model.

Table 3: Nutrition5k Ablation Study. (lower is better)

• MLLM No Context: When food-specific name, density, and bounding box geometry information was removed, food-specific error more than doubled (16.7g → 34.4g, +106%). This clearly demonstrates the importance of the prior information provided by MLLM. Errors increased significantly, especially for plates with multiple foods.
• No structured ownership: When exclusive patch partitioning was replaced with direct mass regression, the error increased by 1.6g (+10%).
• No image-only height: When per-patch height was set to 1 and the volume was reduced to the footprint area, the error increased by 1.2g (+7%).

We can see that food-specific contextual information is the most important component of MLLM, and structured ownership partitioning and learned image-specific height fields also consistently contribute to performance improvement.

4.5. MLLM Flagship Model Comparison 🆚

We benchmarked five leading MLLMs as direct meal size estimators. Each model performed food recognition and estimated grams for each food with an open vocabulary using the same decomposition prompt and matcher. This evaluation was performed on 100 multi-food (≥2 significant foods) plates from the Nutrition5k test set.

Table 4: Flagship MLLM as a direct meal estimator. (Higher mass coverage and lower food-specific MAE are better.)

The three main observations are:

1. Recognition abilities are similar across the board. In the difficult cases, mass-weighted coverage was concentrated in the 61-72% range for all five models, with Claude-Fable-5 being the highest at 71.6%.
2. Differences occur in meal size estimates. The Gemini model and GPT-5.5 were in the 24.1g–26.7g range, while the two Claude models were closer to 36g. Claude models were less prone to underestimate large quantities, but had larger errors for common quantities.
3. Small trained head outperforms all flagship models! Inputting recognition results from Gemini-3.5-Flash, our model achieved an MAE of 16.5g per food on the same 100 plates. That's 32% lower than the best flagship direct estimate. 🚀

5. Limitations 🤔

The four main limitations of the current system are (in order of impact):

1. Fundamental ambiguity in single-point meal estimation: Because footprint alone cannot determine the height of a food pile, large amounts of food tend to be systematically underestimated. This is a limitation of the input image itself rather than a characteristic of the head, and is consistently observed across all datasets.
2. No recognition of invisible foods: Ingredients that leave little visual trace, such as cooking oil, sugar, and sauces, cannot be detected by image-based recognizers. However, they account for a disproportionate share of important nutrients in actual mass and nutritional studies.
3. Dataset-specific training: Even though using the same architecture, loss function, and hyperparameters, each benchmark is trained using its own weights. Cross-dataset transfer from a single universal weigher has not yet been demonstrated.
4. Validation of mass, not nutrients: Food-specific mass was designed for downstream nutrient transformation, but that transformation itself was not evaluated in this study.

6. Conclusion and future research directions 💡

Food recognition is now largely a MLLM capability, but food weight estimation is not. All of the flagship MLLMs we tested performed poorly at meal size estimation, and even larger inference models failed to bridge this gap. Our results show that this gap can be bridged not through scale, but through small, task-specific geometric heads.

Using only the structured output of MLLM and fixed DINOv2 features, this head reduced food-specific meal size error by 33–41% compared to MLLM alone, outperformed the direct estimates of all flagship models, and outperformed the original published image-only model for each benchmark. All of this was done in a completely open-vocabulary, depth-free environment, and without touching MLLM. Therefore, recognition and weight estimation can be clearly separated: MLLM provides identification and prior information, and head provides geometric information. 🤝

The end goal is not a better plate estimator, but rather a replacement for the 24-hour dietary recall method actually used in nutrition research. If future research directions are specified based on this goal, the following tasks become clear:

1. From amount served to amount consumed: Recall measures the amount ingested, while imagery measures the amount presented. Addressing this gap requires "leftover subtraction" by taking pictures (or short videos) before and after ingestion and subtracting the amount of food left over. This is a natural extension of the food-specific weigher.
2. From visible food to full intake: Recall The interviewer asks about the invisible ingredients: cooking oil, coffee sugar, mixed sauces, and the composition of the casserole. No image model can see these. By leveraging MLLM's dietary knowledge interactively (simple follow-up questions, recipe inference through food identification), we can transform these invisible ingredient limitations into finite, queryable problems.
3. From plate to everyday meal: The recall method includes all meal occasions. Deployed systems must address missed captures, snacks, and beverages throughout the day with a single universal meter and degrade performance appropriately when images are not available. (The fact that our identical architecture and hyperparameters already work on three different datasets suggests that a single trainable model is possible.)
4. From benchmark matching to clinical validity: Ultimately, to replace recall, recall methods must be validated the way the recall methods themselves were validated: comparisons against biomarkers and controlled-feeding ground truth, assessment of non-gram nutrient intakes, and evaluation of different cuisines (current MLLM coverage results show that underserved foods are still underserved). We believe that the accuracy of food-specific mass estimation techniques demonstrated here is a key component in making this goal realistic. 💪