Abstract

Document structure editing involves manipulating localized textual, visual, and layout components in document images based on the user’s requests. Past works have shown that multimodal grounding of user requests in the document image and identifying the accurate structural components and their associated attributes remain key challenges for this task. To address these, we introduce the DocEdit-v2, a novel framework that performs end-to-end document editing by leveraging Large Multimodal Models (LMMs). It consists of three novel components: (1) Doc2Command to simultaneously localize edit regions of interest (RoI) and disambiguate user edit requests into edit commands. (2) LLM-based Command Reformulation prompting to tailor edit commands originally intended for specialized software into edit instructions suitable for generalist LMMs. (3) Moreover, DocEdit-v2 processes these outputs via Large Multimodal Models like GPT-4V and Gemini, to parse the document layout, execute edits on grounded Region of Interest (RoI), and generate the edited document image. Extensive experiments on the DocEdit dataset show that DocEdit-v2 significantly outperforms strong baselines on edit command generation (2-33%), RoI bounding box detection (12-31%), and overall document editing (1-12%) tasks.

The problem

Language-guided document editing takes a document image and a request like “make the numbered items 9 and 9A into sub-headings a and b” and has to produce the edited document. Three things make this hard: the request has to be grounded to the right region of a dense page, the request has to be turned into a precise action on a specific component with specific attributes, and the edit has to be applied without disturbing the rest of the document.

Earlier work generated software-specific edit commands but stopped short of producing an edited document, and pixel-level generative models struggle to reproduce text-dense pages faithfully. Large multimodal models are good at layout parsing and code synthesis, but handing them a raw request and a page image leaves the grounding and disambiguation to chance.

Approach

DocEdit-v2 treats document editing as editing the document’s HTML+CSS representation, which preserves hierarchy and separates content from style, and it builds the prompt for that edit in three stages.

  • Doc2Command. A multi-task multimodal Transformer that performs region-of-interest segmentation and edit command generation jointly. The user request is rendered as a text box on top of the document image, and the combined image is encoded by a Pix2Struct ViT encoder with variable-resolution patching. A text decoder generates the structured command ACTION(<Component>, <Initial State>, <Final State>), and a DETR-style mask transformer predicts a three-class segmentation map (region of interest, rendered request, rest of the page) that is converted to a bounding box at inference. Training uses a weighted sum of the text loss and a focal-plus-dice segmentation loss. Doc2Command is described in more detail on its own page, Doc2Command.
  • Command Reformulation prompting. Commands generated from the DocEdit dataset are written for specialized editing software and often underspecify the action or component. An LLM (GPT-4 or Gemini Pro) rewrites the generated command, given the original request, into an instruction suitable for a generalist multimodal model.
  • Generative document editing. GPT-4V or Gemini receives the document image with the grounded bounding box drawn as a set-of-marks cue, the reformulated instruction, and the document’s HTML+CSS, and produces the edited HTML+CSS. Both input and ground-truth documents are converted to HTML+CSS with constrained prompts (standard class names, flexbox layout, embedded CSS, placeholders for media) so that outputs can be compared fairly.
Full DocEdit-v2 pipeline on a real example. The request to convert items 9 and 9A into a main heading with sub-headings is grounded to the right rows, turned into a REPLACE command, reformulated as a modify-text instruction, and applied to produce the edited HTML+CSS document.

We also introduce two automated metrics for HTML document editing, DOM Tree Edit Distance (Zhang-Shasha) and CSS IoU over property-value pairs, alongside a three-part human evaluation of Style Replication, Content Replication, and Edit Correctness. CSS IoU correlates at 0.73 with human Style Replication judgments.

Document image User request "make items 9 and 9A sub-headings a, b" Doc2Command request rendered onto the page Grounded RoI segmentation map → bounding box Edit command REPLACE(bullet, 9 and 9A, a and b) Reformulation GPT-4 / Gemini Pro rewrite as instruction GPT-4V / Gemini page + RoI marks + instruction + HTML+CSS edited HTML+CSS set-of-marks cue original request
DocEdit-v2: Doc2Command grounds the request and emits a structured command, an LLM reformulates the command into an instruction, and GPT-4V or Gemini edits the page's HTML+CSS with the grounded region marked.

Example

The paper’s Figure 1 (shown above) walks one request through the whole pipeline. Each stage’s output is transcribed below.

User request + document image

Text "Delta Electricity" with the numbering 9 and 9A are converted to main heading "9" and sub heading to "a" and "b" (on a "Register of Facility Providers" page listing numbered corporations).

Doc2Command output

Visual grounding: a bounding box around list items 9 (Delta Electricity) and 9A (Duke Energy Australia).
Edit command: REPLACE(BULLET, BULLETS AS 9 AND 9A, BULLETS AS A AND B)

Reformulated command

modify(text, "Delta Electricity" with numbering 9 and 9A, main heading "9" and sub headings "a" and "b")

Edited HTML+CSS document

Item 9 becomes a heading with two sub-items: "9. a. Delta Electricity (being a corporation established under the Energy Services Corporations Act 1995 (NSW)) b. Duke Energy Australia Trading and marketing A.C.N. 063 050 168 Pty Limited", and the rest of the list is unchanged.

The second example (Figure 3 of the paper) shows why Command Reformulation matters: the raw Doc2Command command is correct but underspecified, and the generalist model edits the wrong “22”.

Figure 3 of the paper. Left: the document edited from the raw Doc2Command command. Centre: the request and page. Right: the document edited from the reformulated command.

User request

Changed the page number from 22 to 202.

Doc2Command command

MODIFY(TEXT, 22, 202) — the edited document targets the "(22)" operating-cost cell in the table and the footer still reads 22.

Reformulated command

MODIFY(TEXT, page number 22, page number 202) — the edited document's footer reads "page number 202".

Results

All experiments use the DocEdit-PDF dataset (17,808 document image pairs with edit requests and ground-truth commands), evaluated on the official test split. The main result is the end-to-end editing ablation with GPT-4V as the editing model.

Setting (GPT-4V as editor) Tree Edit Distance CSS IoU Edit Correctness (%) Human total (%)
GPT-4V only: raw request + page 24.13 0.245 27.45 55.92
+ visual grounding 24.02 0.250 45.28 62.59
+ command generation 23.54 0.247 49.32 64.19
+ command generation + reformulation 23.27 0.245 51.87 65.49
+ visual grounding + command generation 23.72 0.251 55.33 66.79
DocEdit-v2: grounding + command + reformulation 23.15 0.252 57.41 67.28

Source: Table 3 of the paper. Tree Edit Distance is the Zhang-Shasha distance between DOM trees (lower is better); CSS IoU is over property-value pairs, Edit Correctness is the human rating of whether the requested edit was made, and Human total averages Style Replication, Content Replication, and Edit Correctness (higher is better).

Setting (Gemini as editor) Tree Edit Distance CSS IoU Edit Correctness (%) Human total (%)
Gemini only: raw request + page 62.95 0.333 15.79 45.61
+ visual grounding 54.63 0.332 39.22 54.79
+ command generation + reformulation 53.89 0.341 40.44 56.69
DocEdit-v2: grounding + command + reformulation 52.24 0.367 44.73 58.77

Source: Table 4 of the paper, same metrics.

  • Grounding plus reformulation raises GPT-4V’s Edit Correctness by 29.96 points (27.45 to 57.41) and its overall human score by 11.36 points; with Gemini the gains are 28.94 and 13.16 points.
  • Visual grounding alone is the single biggest lever (+17.83 Edit Correctness for GPT-4V, +23.43 for Gemini); Command Reformulation adds a further 2-3 points on top of the generated command.
  • Command generation (Table 1). Doc2Command reaches 39.6% exact match, 0.86 ROUGE-L, and 86.1% component accuracy, against 37.6%, 0.83, and 40.7% for DocEditor and 14.3%, 0.78, and 75.03% for GPT-4 with in-context examples.
  • RoI bounding box detection (Table 2). Doc2Command achieves 48.69% Top-1 accuracy, ahead of DocEditor at 36.50%, TransVG at 25.34%, and ReSC-Large at 17.04%. Both tables are reproduced on the Doc2Command page.
  • CSS IoU correlates at 0.73 with human Style Replication judgments, which is why the automated metrics track the human ones in the tables above.

Resources