Spaces:

WebShepherd
/

Project-Web-Shepherd

Running

App Files Files Community

WebShepherd commited on May 16

Commit

25c711a

verified ·

1 Parent(s): cf9ac4d

Update index.html

Browse files

Files changed (1) hide show

index.html +10 -10

index.html CHANGED Viewed

@@ -3,10 +3,10 @@
 <head>
   <meta charset="utf-8">
   <meta name="description"
-        content="WEB-SHEPHERD: Advancing PRMs for Reinforcing Web Agents">
   <meta name="keywords" content="Nerfies, D-NeRF, NeRF">
   <meta name="viewport" content="width=device-width, initial-scale=1">
-  <title>WEB-SHEPHERD: Advancing PRMs for Reinforcing Web Agents</title>
   <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
         rel="stylesheet">
@@ -35,7 +35,7 @@
         <div class="column has-text-centered">
           <h1 class="title is-1 publication-title">
             <img src="static/images/shepherd_emoji.png" style="width:1em;vertical-align: middle" alt="Logo"/>
-            WEB-SHEPHERD:
           </h1>
           <h2 class="subtitle is-3 publication-subtitle">
             Advancing PRMs for Reinforcing Web Agents
@@ -141,7 +141,7 @@
   <div class="container is-max-desktop">
         <div class="content has-text-centered">
           <img src="static/images/figure_1.png" alt="geometric reasoning" width="95%"/>
-          <p> Performance and cost-efficiency of WEB-SHEPHERD (3B). WEB-SHEPHERD achieves the state-of-the-art performance while requiring significantly lower cost compared to existing baselines. </p>
         </div>
       <!-- </div> -->
     </div>
@@ -160,10 +160,10 @@
           </p>
           <p>
             Yet, specialized reward models for web navigation that can be utilized during both training and test-time have been absent until now. Despite the importance of speed and cost-effectiveness, prior works have utilized MLLMs as reward models, which poses significant constraints for real-world deployment.
-            To address this, in this work, we propose the first process reward model (PRM) called WEB-SHEPHERD which could assess web navigation trajectories in a step-level. To achieve this, we first construct the WEBPRM collection, a large-scale dataset with 40K step-level preference pairs and annotated checklists spanning diverse domains and difficulty levels. Next, we also introduce the WEB-RewardBench, the first meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe that our WEB-SHEPHERD achieves about 30 points better accuracy compared to using GPT-4o on WEB-RewardBench.
           </p>
           <p>
-            Furthermore, when testing on WebArena-lite by using GPT-4o-mini as the policy and WEB-SHEPHERD as the verifier, we achieve 10.3 points better performance, in 10 times less cost compared to using GPT-4o-mini as the verifier.
           </p>
         </div>
       </div>
@@ -207,7 +207,7 @@
 <section class="hero is-light is-small">
   <div class="hero-body has-text-centered">
   <h1 class="title is-1 mmmu">
-    <span class="mmmu" style="vertical-align: middle">WEB-SHEPHERD</span>
   </h1>
   </div>
 </section>
@@ -224,10 +224,10 @@
         </div>
         <div class="content has-text-justified">
           <p>
-            We introduce WEB-SHEPHERD, a process reward model designed to provide dense and reliable supervision to web agents and enable more informative credit assignment.
           </p>
           <p>
-            We train WEB-SHEPHERD on the WEBPRM Collection to support two key functionalities: (1) generating task-specific checklists, and (2) assigning rewards based on checklist completion.
           </p>
         </div>
     </div>
@@ -258,7 +258,7 @@
             <div class="content has-text-justified">
               <p>
-                Table above reports the evaluation results on WEB-RewardBench. As shown in Table, state-of-the-art MLLMs struggle to provide reliable rewards for web navigation tasks. This limitation is particularly evident in the trajectory accuracy metric. In this measure, models frequently fail to assign correct rewards consistently at each time step within a single task. In contrast, WEB-SHEPHERD significantly outperforms all baselines, demonstrating a substantial performance gap across all benchmark settings.
               </p>
               <p>
                 Also, Table above demonstrates that both baseline and our models benefit significantly from the checklist in assigning rewards.

 <head>
   <meta charset="utf-8">
   <meta name="description"
+        content="Web-Shepherd: Advancing PRMs for Reinforcing Web Agents">
   <meta name="keywords" content="Nerfies, D-NeRF, NeRF">
   <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>Web-Shepherd: Advancing PRMs for Reinforcing Web Agents</title>
   <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
         rel="stylesheet">
         <div class="column has-text-centered">
           <h1 class="title is-1 publication-title">
             <img src="static/images/shepherd_emoji.png" style="width:1em;vertical-align: middle" alt="Logo"/>
+            Web-Shepherd:
           </h1>
           <h2 class="subtitle is-3 publication-subtitle">
             Advancing PRMs for Reinforcing Web Agents
   <div class="container is-max-desktop">
         <div class="content has-text-centered">
           <img src="static/images/figure_1.png" alt="geometric reasoning" width="95%"/>
+          <p> Performance and cost-efficiency of Web-Shepherd (3B). Web-Shepherd achieves the state-of-the-art performance while requiring significantly lower cost compared to existing baselines. </p>
         </div>
       <!-- </div> -->
     </div>
           </p>
           <p>
             Yet, specialized reward models for web navigation that can be utilized during both training and test-time have been absent until now. Despite the importance of speed and cost-effectiveness, prior works have utilized MLLMs as reward models, which poses significant constraints for real-world deployment.
+            To address this, in this work, we propose the first process reward model (PRM) called Web-Shepherd which could assess web navigation trajectories in a step-level. To achieve this, we first construct the WEBPRM collection, a large-scale dataset with 40K step-level preference pairs and annotated checklists spanning diverse domains and difficulty levels. Next, we also introduce the WEB-RewardBench, the first meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe that our Web-Shepherd achieves about 30 points better accuracy compared to using GPT-4o on WEB-RewardBench.
           </p>
           <p>
+            Furthermore, when testing on WebArena-lite by using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve 10.3 points better performance, in 10 times less cost compared to using GPT-4o-mini as the verifier.
           </p>
         </div>
       </div>
 <section class="hero is-light is-small">
   <div class="hero-body has-text-centered">
   <h1 class="title is-1 mmmu">
+    <span class="mmmu" style="vertical-align: middle">Web-Shepherd</span>
   </h1>
   </div>
 </section>
         </div>
         <div class="content has-text-justified">
           <p>
+            We introduce Web-Shepherd, a process reward model designed to provide dense and reliable supervision to web agents and enable more informative credit assignment.
           </p>
           <p>
+            We train Web-Shepherd on the WEBPRM Collection to support two key functionalities: (1) generating task-specific checklists, and (2) assigning rewards based on checklist completion.
           </p>
         </div>
     </div>
             <div class="content has-text-justified">
               <p>
+                Table above reports the evaluation results on WEB-RewardBench. As shown in Table, state-of-the-art MLLMs struggle to provide reliable rewards for web navigation tasks. This limitation is particularly evident in the trajectory accuracy metric. In this measure, models frequently fail to assign correct rewards consistently at each time step within a single task. In contrast, Web-Shepherd significantly outperforms all baselines, demonstrating a substantial performance gap across all benchmark settings.
               </p>
               <p>
                 Also, Table above demonstrates that both baseline and our models benefit significantly from the checklist in assigning rewards.