Abstract

This short paper introduces Doc2Command, a multi-task multimodal model for language-guided document editing. Given a document image and a natural-language edit request, Doc2Command jointly localizes the region of interest that the request refers to and translates the request into a structured edit command with an action, a component, and its initial and final states. The request is rendered directly onto the document image so that a single image encoder processes language and layout together; a text decoder generates the command and a mask transformer produces the grounding as a segmentation map. Doc2Command later became the grounding and command generation component of DocEdit-v2 (EMNLP 2024), and this summary is based on the description of the model in that paper.

The problem

Edit requests in the DocEdit dataset are open-vocabulary and often ambiguous: “change the page number in the footer from 11 to 12” has to be resolved to a specific component on a dense page and to a precise action on that component. Prior work treated grounding and command generation as separate problems, and models that regress bounding box coordinates directly perform poorly on document images, where the target may be a single line of text among hundreds.

Approach

  • One visual input. Rather than encoding the request as text, we render it as a text box on top of the document image and feed the combined image to a pre-trained Vision Transformer encoder borrowed from Pix2Struct. The image is not scaled to a fixed resolution; the scaling factor is chosen to fit as many fixed-size patches as the encoder’s sequence length allows, which keeps the model robust to extreme document aspect ratios.
  • Edit command generation. A pre-trained Pix2Struct text decoder autoregressively generates the command ACTION(<Component>, <Initial State>, <Final State>) from the patch embeddings.
  • Multimodal grounding as segmentation. A DETR-style mask transformer with three learnable class embeddings (region of interest, rendered request, remaining document) produces class masks by a scalar product with the patch embeddings. The masks are upsampled to image size, softmaxed, and converted at inference into a bounding box from the largest contiguous region around the mask centroid.
  • Training. The decoder and mask transformer are fine-tuned jointly with a weighted loss, a text loss for the command and a segmentation loss that sums focal and dice loss.
Document image User request "change page number from 22 to 202" Render request as a text box on top of the page Pix2Struct ViT encoder variable-resolution patches Text decoder autoregressive command tokens Mask transformer 3 classes: RoI, request, rest of page Edit command MODIFY(text, 22, 202) Grounded RoI seg. map upsampled → bounding box one visual input joint loss: text + focal + dice
Doc2Command: the request is rendered onto the page and encoded once; a text decoder writes the command and a mask transformer segments the region of interest, which is converted to a bounding box.

Example

The end-to-end example from the DocEdit-v2 paper (its Figure 1) shows what Doc2Command produces for one request before the later stages take over.

Figure 1 of the DocEdit-v2 paper. Doc2Command takes the document image and request and returns the grounded region (items 9 and 9A) and the structured edit command; the later stages reformulate the command and edit the HTML+CSS.

Input: document image + user request

Text "Delta Electricity" with the numbering 9 and 9A are converted to main heading "9" and sub heading to "a" and "b"

Output: grounding + edit command

Region of interest: a bounding box around list items 9 (Delta Electricity) and 9A (Duke Energy Australia).
Command: REPLACE(BULLET, BULLETS AS 9 AND 9A, BULLETS AS A AND B)

Table 5 of the DocEdit-v2 paper lists generated commands next to the ground truth. Three of them:

Request

Added page number 4 at the footer of the page.

Predicted vs. ground truth

add(text footer, none, Page 4) — matches the ground truth exactly.

Request

Change the date "December 1, 2000" to December 11, 2020

Predicted vs. ground truth

replace(text, December 1, 2000, December, 11, 2000) vs. ground truth modify(text, 1, 2000, 11, 2000) — the generated command achieves the desired edit, but the ground truth does it with fewer changes.

Request

2-3 lines of text in the paragraph "(p) Issues, obtain" are changed to four separate bullet points. ...

Predicted vs. ground truth

replace(bullet, dotted, 4 bullet points) vs. ground truth split(text, paragraph, split) — the model mistakes a "split" action for a "replace" action.

Results

The numbers below are reported in the DocEdit-v2 paper, which evaluates Doc2Command on the test split of the DocEdit-PDF dataset (17,808 document image pairs with edit requests and ground-truth commands).

System Exact match (%) Word overlap F1 ROUGE-L Action (%) Component (%)
T5 (text only) 20.4 0.79 0.76 81.4 29.8
Multimodal Transformer 31.6 0.82 0.83 83.1 32.4
DocEditor 37.6 0.87 0.83 87.6 40.7
GPT-3.5 (in-context) 10.1 0.77 0.77 75.93 73.37
GPT-4 (in-context) 14.3 0.78 0.78 81.57 75.03
Doc2Command 39.6 0.87 0.86 85.0 86.1

Source: Table 1 of the DocEdit-v2 paper, edit command generation. Action and Component are the accuracy of the predicted action and component fields. Higher is better.

System Top-1 RoI accuracy (%)
ReSC-Large 17.04
TransVG 25.34
DocEditor 36.50
Doc2Command 48.69

Source: Table 2 of the DocEdit-v2 paper, region-of-interest bounding box detection. Higher is better.

  • Doc2Command has the best exact match and ROUGE-L, and its component accuracy (86.1%) is more than 45 points above the task-specific DocEditor and about 11 points above GPT-4, which identifies components well but rarely reproduces the exact command.
  • Treating grounding as segmentation rather than coordinate regression lifts Top-1 RoI accuracy by 12.19 points over DocEditor and by more than 23 points over the direct-regression baselines.
  • Within DocEdit-v2, plugging Doc2Command’s grounding and command into the GPT-4V editing prompt raises Edit Correctness from 27.45% to 55.33% (Table 3 of that paper).

Resources