Create README.md

README.md

@@ -0,0 +1,97 @@
+---
+license: cc
+datasets:
+- weizhiwang/Open-Qwen2VL-Data
+- MAmmoTH-VL/MAmmoTH-VL-Instruct-12M
+language:
+- en
+base_model:
+- Qwen/Qwen2.5-1.5B-Instruct
+- google/siglip-so400m-patch14-384
+pipeline_tag: image-text-to-text
+---
+
+# Model Card for Open-Qwen2VL
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+<!-- Please follow my reproduced implementation [LLaVA-Unified](https://github.com/Victorwz/LLaVA-Unified) for more details on fine-tuning LLaVA model with Llama-3 as the foundatiaon LLM. -->
+
+## Updates 
+<!-- - [5/14/2024] The codebase has been upgraded to llava-next (llava-v1.6). Now it supports the latest llama-3, phi-3, mistral-v0.1-7b models. -->
+
+## Model Details
+<!-- Follows LLavA-1.5 pre-train and supervised fine-tuning pipeline. You do not need to change the LLaVA codebase to accommodate Llama-3.  -->
+
+## How to Use
+
+Please firstly install Open-Qwen2VL via
+```
+pip install git+https://github.com/Victorwz/Open-Qwen2VL.git
+```
+
+You can load the model and perform inference as follows:
+```python
+import requests
+import torch
+from PIL import Image
+from prismatic import load
+
+device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
+
+# Load a pretrained VLM (either local path, or ID to auto-download from the HF Hub)
+vlm = load("Open-Qwen2VL")
+vlm.to(device, dtype=torch.bfloat16)
+
+# Download an image and specify a prompt
+image_url = "https://huggingface.co/adept/fuyu-8b/resolve/main/bus.png"
+# image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
+image = [vlm.vision_backbone.image_transform(Image.open(requests.get(image_url, stream=True).raw).convert("RGB")).unsqueeze(0)]
+user_prompt = '<image>' + '\n' + "Describe the image."
+
+# Generate!
+generated_text = vlm.generate_batch(
+    image,
+    [user_prompt],
+    do_sample=False,
+    max_new_tokens=512,
+    min_length=1,
+)
+print(generated_text[0])
+```
+The image caption results look like:
+```
+The image depicts a blue and orange bus parked on the side of a street. The bus is a modern, single-decker model, and it is prominently displayed in the foreground. The bus has a digital display at the front, showing the route number "616" and the destination "Morton Best." The bus is labeled with the logo of "Stagecoach," a well-known bus operator in the UK. The bus also features a slogan on its side that reads "Stronger for you," indicating a focus on providing reliable and efficient public transportation.
+
+The bus is parked on a grassy area adjacent to a sidewalk. The grass is well-maintained, and there are a few trees planted along the sidewalk, providing some shade. The sidewalk itself is made of concrete and appears to be clean and well-kept. 
+
+In the background, there are residential buildings, which are typical of a suburban area. These buildings have pitched roofs and are constructed with a mix of brick and plaster. The architecture suggests a typical British suburban neighborhood. The sky above is partly cloudy, with patches of blue sky visible, indicating fair weather.
+
+The bus is parked in a designated bus stop area, as indicated by the presence of a bus stop sign and a bus stop shelter, although the shelter is not visible in the image. The bus's license plate is visible and reads "Y600 HJX." The overall scene suggests a typical day in a suburban area where public transportation is readily available for residents.
+
+The image does not contain any people, vehicles other than the bus, or any other notable objects. The focus is primarily on the bus and its surroundings. The bus appears to be in good condition, indicating that it is well-maintained and likely in regular use. The presence of the bus stop and the well-kept environment suggest that the area is well-organized and that public transportation is an integral part of the community's daily life.
+```
+
+<!-- # Fine-Tune LLaVA-Llama-3 on Your Visual Instruction Data
+Please refer to our [LLaVA-Unified](https://github.com/Victorwz/LLaVA-Unified) git repo for fine-tuning data preparation and scripts. The data loading function and fastchat conversation template are changed due to a different tokenizer.
+
+## Benchmark Results
+
+
+| Model                 |      MMMU Val     |
+| :-------------------- | :---------------: |
+| LLaVA-v1.5-7B         |       35.3        |
+| LLaVA-Llama-3-8B      |       36.7        | 
+
+Please refer to `eval_outputs/LLaVA-Llama-3-8B_mmmu_val.json` for reproduce the benchmark performance on MMMU validation set. -->
+
+## Citation
+<!-- 
+```bibtex
+@misc{wang2024llavallama3,
+  title={LLaVA-Llama-3-8B: A reproduction towards LLaVA-v1.5 based on Llama-3-8B LLM backbone},
+  author={Wang, Weizhi},
+  year={2024}
+}
+``` -->