{
  "name": "SocialCrawl: Crawl a site and get each page as markdown",
  "nodes": [
    {
      "id": "node-1",
      "position": [
        260,
        0
      ],
      "name": "Run recipe",
      "type": "n8n-nodes-base.manualTrigger",
      "typeVersion": 1,
      "parameters": {}
    },
    {
      "id": "node-2",
      "position": [
        520,
        0
      ],
      "name": "crawl",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "parameters": {
        "method": "POST",
        "url": "https://www.socialcrawl.dev/v1/web/crawl",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "sendBody": true,
        "contentType": "json",
        "specifyBody": "json",
        "jsonBody": "{\n  \"url\": \"https://docs.example.com/\",\n  \"limit\": \"50\",\n  \"formats\": \"markdown\"\n}",
        "options": {
          "timeout": 300000
        }
      },
      "credentials": {
        "httpHeaderAuth": {
          "name": "SocialCrawl API key"
        }
      },
      "executeOnce": true
    },
    {
      "id": "node-3",
      "position": [
        780,
        0
      ],
      "name": "result",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "parameters": {
        "method": "GET",
        "url": "=https://www.socialcrawl.dev/v1/web/jobs/{{ $('crawl').first().json.data.job_id }}",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "options": {
          "timeout": 300000
        }
      },
      "credentials": {
        "httpHeaderAuth": {
          "name": "SocialCrawl API key"
        }
      },
      "executeOnce": true,
      "retryOnFail": true,
      "maxTries": 4,
      "waitBetweenTries": 2000
    },
    {
      "id": "node-4",
      "position": [
        1040,
        0
      ],
      "name": "result: finished?",
      "type": "n8n-nodes-base.if",
      "typeVersion": 2,
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "leftValue": "",
            "typeValidation": "loose"
          },
          "conditions": [
            {
              "id": "result-finished",
              "leftValue": "={{ !['queued','running','pending'].includes($json.data.status) }}",
              "rightValue": true,
              "operator": {
                "type": "boolean",
                "operation": "true",
                "singleValue": true
              }
            }
          ],
          "combinator": "and"
        },
        "options": {}
      }
    },
    {
      "id": "node-5",
      "position": [
        1300,
        0
      ],
      "name": "result: wait",
      "type": "n8n-nodes-base.wait",
      "typeVersion": 1.1,
      "parameters": {
        "resume": "timeInterval",
        "amount": 5,
        "unit": "seconds"
      }
    },
    {
      "id": "node-6",
      "position": [
        1560,
        0
      ],
      "name": "result: result",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "parameters": {
        "jsCode": "// The finished answer.\nreturn $input.all();"
      }
    },
    {
      "id": "note",
      "name": "How this recipe works",
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        0,
        -460
      ],
      "parameters": {
        "content": "## Crawl a site and get each page as markdown\n\nPrice for fifty pages: 50 credits held up front (settles 1-50; the unused hold is refunded).\n\nMeasured on 25/10/2026: 50 credits in 9 calls for ITEMS=25 (the quote held 50). The TypeScript and Python programs both ran.\n\nFRESH=1 skips the cache and bills every call; a cache hit is free.\n\nInputs, set to a working example (change them to yours):\n\n  site_url: Root URL to crawl\n\n  max_pages: Most pages to crawl\n\n**Key:** create a Header Auth credential named SocialCrawl API key, with Name x-api-key and your API key as Value, and pick it on every request node.\n\n**Rows:** a step that runs once per row takes the first 50 rows (change the number in its rows node).\n\n**Errors:** a request node stops the workflow on an API error. Read error.type in its output; retry only when error.retryable is true.",
        "width": 520,
        "height": 420
      }
    }
  ],
  "connections": {
    "Run recipe": {
      "main": [
        [
          {
            "node": "crawl",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "crawl": {
      "main": [
        [
          {
            "node": "result",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "result": {
      "main": [
        [
          {
            "node": "result: finished?",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "result: finished?": {
      "main": [
        [
          {
            "node": "result: result",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "result: wait",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "result: wait": {
      "main": [
        [
          {
            "node": "result",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  },
  "settings": {}
}
