Install Self-Hosted
Get Started on Cloud
BasicsPipelinesActions & ServicesEnvironmentsIntegrationsTargetsTestsDomainsSandboxesAgents & Tunnels

Scrape pages

View as Markdown
  • POST
  • /workspaces
  • /:workspace
  • /projects
  • /:project_name
  • /pipelines
  • /:pipeline_id
  • /actions

Creates a new Scrape pages action in the pipeline

Request

REQUIRED SCOPES
EXECUTION_MANAGE
URL PARAMETERS
workspacerequiredstring
The human-readable ID of the workspace

Example: "my-company"

project_namerequiredstring
The human-readable ID of the project

Example: "my-project"

pipeline_idrequiredinteger
The ID of the pipeline

Example: 123

POST PARAMETERS
typerequiredstring enum
The type of the action
Allowed enum:
SCRAPE
urlstring
URL to scrape (optional; defaults to scrape suite URL)
html_urlread-onlystring
Web URL to view this object in Buddy.works
after_action_idinteger
The numerical ID of the action, after which this action should be added
namerequiredstring
The name of the action
trigger_timestring enum
Specifies when the action should be executed
Allowed enum:
ON_EVERY_EXECUTION,
ON_SUCCESS,
ON_FAILURE,
ON_BACK_TO_SUCCESS,
ON_WARNING,
ON_WAIT_FOR_APPROVE,
ON_TERMINATE
trigger_conditionsTriggerConditionView[]
The list of trigger conditions to meet so that the action can be triggered
run_nextstring enum
Defines whether the action should run in parallel with the next one
Allowed enum:
WAIT_ON_SUCCESS,
IN_SOFT_PARALLEL,
IN_HARD_PARALLEL
run_only_on_first_failureboolean
Defines whether the action should be executed on each failure. Restricted to and required if the trigger_time is ON_FAILURE
variablesEnvironmentVariableView[]
The list of variables you can use the action
disabledboolean
When set to true the action is disabled. By default it is set to false
timeoutinteger
The timeout in seconds
ignore_errorsboolean
If set to true the execution will proceed, mark action as a warning and jump to the next action. Doesn't apply to deployment actions
retry_countinteger
Number of retries if the action fails
retry_intervalinteger
Delay time between auto retries in seconds
loopstring[]
The list of commands that will be executed in a loop
current_revisionstring
The git revision of the last successful run of the action
permissionsPermissionsView
Access permissions configuration
scrape_suitestring
The scrape suite identifier
followboolean
Scrape all subviews of the page. Default: false
respect_robotsboolean
Respect robots.txt. Default: false
outputsScrapeOutputView[]
Output types to produce (type, selector, quality, fullPage)
browsersstring[]
Browsers to use (e.g., CHROMIUM, FIREFOX, WEBKIT)
color_schemestring
Color scheme to use (LIGHT, DARK, LIGHT_AND_DARK)
delaysstring[]
Add delay in milliseconds before taking screenshot (format: [scope::]milliseconds, e.g., "1000" or "example.com::2000")
wait_forstring[]
Wait for elements to appear before scraping (format: [scope::]type=value, e.g., "CSS=#content" or "example.com::XPATH=//div[@class='loaded']")
local_storagestring[]
Set localStorage values used when visiting URLs (format: [scope::]key=value)
devicesScrapeDeviceView[]
Device definitions (name, viewport, screen, devicePixelRatio, isMobile)
output_directorystring
Output directory for scrape results. Default: "."
cookiesstring[]
Set cookies used when visiting the URLs (format: [scope::]cookie_value, e.g., "session=abc123" or "example.com::auth=token123; Path=/; Secure; HttpOnly")
headersstring[]
Set HTTP headers used when visiting the URLs (format: [scope::]name=value, e.g., "Authorization=Bearer token" or "example.com::Accept=application/json")

Response

RESPONSE BODY
typestring enum
The type of the action
Allowed enum:
SCRAPE
urlstring
URL to scrape (optional; defaults to scrape suite URL)
html_urlread-onlystring
Web URL to view this object in Buddy.works
idinteger
The ID of the action
namestring
The name of the action
current_revisionstring
The git revision of the last successful run of the action
trigger_timestring enum
Specifies when the action should be executed
Allowed enum:
ON_EVERY_EXECUTION,
ON_SUCCESS,
ON_FAILURE,
ON_BACK_TO_SUCCESS,
ON_WARNING,
ON_WAIT_FOR_APPROVE,
ON_TERMINATE
trigger_conditionsTriggerConditionView[]
The list of trigger conditions to meet so that the action can be triggered
run_nextstring enum
Defines whether the action should run in parallel with the next one
Allowed enum:
WAIT_ON_SUCCESS,
IN_SOFT_PARALLEL,
IN_HARD_PARALLEL
run_only_on_first_failureboolean
Defines whether the action should be executed on each failure. Restricted to and required if the trigger_time is ON_FAILURE
variablesEnvironmentVariableView[]
The list of variables you can use the action
disabledboolean
When set to true the action is disabled. By default it is set to false
timeoutinteger
The timeout in seconds
ignore_errorsboolean
If set to true the execution will proceed, mark action as a warning and jump to the next action. Doesn't apply to deployment actions
retry_countinteger
Number of retries if the action fails
retry_intervalinteger
Delay time between auto retries in seconds
last_execution_statusstring enum
The status of the last run of the action
Allowed enum:
INPROGRESS,
ENQUEUED,
TERMINATED,
SUCCESSFUL,
FAILED,
INITIAL,
NOT_EXECUTED,
SKIPPED,
TERMINATING,
WAITING_FOR_APPLY,
WAITING_FOR_VARIABLES,
WAITING_FOR_SETTABLE_VARIABLES,
WAITING_FOR_VT_SESSION
pipelineShortPipelineView
Short representation of a pipeline
permissionsPermissionsView
Access permissions configuration
loopstring[]
The list of commands that will be executed in a loop
scrape_suitestring
The scrape suite identifier
followboolean
Scrape all subviews of the page. Default: false
respect_robotsboolean
Respect robots.txt. Default: false
outputsScrapeOutputView[]
Output types to produce (type, selector, quality, fullPage)
browsersstring[]
Browsers to use (e.g., CHROMIUM, FIREFOX, WEBKIT)
color_schemestring
Color scheme to use (LIGHT, DARK, LIGHT_AND_DARK)
delaysstring[]
Add delay in milliseconds before taking screenshot (format: [scope::]milliseconds, e.g., "1000" or "example.com::2000")
wait_forstring[]
Wait for elements to appear before scraping (format: [scope::]type=value, e.g., "CSS=#content" or "example.com::XPATH=//div[@class='loaded']")
local_storagestring[]
Set localStorage values used when visiting URLs (format: [scope::]key=value)
devicesScrapeDeviceView[]
Device definitions (name, viewport, screen, devicePixelRatio, isMobile)
output_directorystring
Output directory for scrape results. Default: "."
cookiesstring[]
Set cookies used when visiting the URLs (format: [scope::]cookie_value, e.g., "session=abc123" or "example.com::auth=token123; Path=/; Secure; HttpOnly")
headersstring[]
Set HTTP headers used when visiting the URLs (format: [scope::]name=value, e.g., "Authorization=Bearer token" or "example.com::Accept=application/json")

Last modified on Mar 18, 2026

Previous pageEdit actionNext pageGet action

Request example

curl -X POST "https://api.buddy.works/workspaces/:workspace/projects/:project_name/pipelines/:pipeline_id/actions" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
  "name": "Scrape",
  "type": "SCRAPE",
  "trigger_time": "ON_EVERY_EXECUTION",
  "scrape_suite": "my-scrape-suite",
  "url": "https://example.com",
  "follow": true,
  "respect_robots": true,
  "outputs": [
    {
      "type": "PNG",
      "quality": 90,
      "full_page": true
    }
  ],
  "browsers": [
    "chromium"
  ],
  "color_scheme": "dark",
  "cookies": [
    "session=abc123"
  ],
  "headers": [
    "Authorization: Bearer token"
  ],
  "delays": [
    "1000"
  ],
  "wait_for": [
    "#main-content"
  ],
  "local_storage": [
    "key=value"
  ],
  "output_directory": "/scrape-output"
}'
STATUS
200 OK