Form Data and Multipart

Two flavors of form posting. Pick based on what you're shipping:

application/x-www-form-urlencoded for plain key/value text fields. Small, simple, no binary.
multipart/form-data for file uploads, mixed text + file fields, or anything binary.

Browsers pick the same way. <form> with enctype="multipart/form-data" for uploads, default url-encoded otherwise.

URL-encoded

The body's just key1=val1&key2=val2, with values percent-encoded. Content-Type is application/x-www-form-urlencoded.

Go
Python
Node.js
.NET

Go gives you net/url.Values to build the body, then you ship the bytes.

package main

import (
    "bytes"
    "context"
    "fmt"
    "net/url"

    httpcloak "github.com/sardanioss/httpcloak"
)

func main() {
    s := httpcloak.NewSession("chrome-latest")
    defer s.Close()

    form := url.Values{}
    form.Set("user", "alice")
    form.Set("token", "abc 123")

    req := &httpcloak.Request{
        Method: "POST",
        URL:    "https://httpbin.org/post",
        Headers: map[string][]string{
            "Content-Type": {"application/x-www-form-urlencoded"},
        },
        Body: bytes.NewReader([]byte(form.Encode())),
    }
    resp, _ := s.Do(context.Background(), req)
    defer resp.Close()

    body, _ := resp.Text()
    fmt.Println(body) // "form": {"user": "alice", "token": "abc 123"}
}

The non-session client has client.PostForm(ctx, url, bodyBytes) too, which sets the Content-Type for you.

import httpcloak

s = httpcloak.Session(preset="chrome-latest")

r = s.post(
    "https://httpbin.org/post",
    data={"user": "alice", "token": "abc 123"},
)

print(r.json()["form"])  # {"user": "alice", "token": "abc 123"}

When data= is a dict, the binding url-encodes it and sets the Content-Type for you. Pass data= a string or bytes and it ships them as-is, with whatever Content-Type you put in headers.

const { Session } = require("httpcloak");

const s = new Session({ preset: "chrome-latest" });

const r = await s.post("https://httpbin.org/post", {
  form: { user: "alice", token: "abc 123" },
});

console.log(r.json().form);

using HttpCloak;

using var s = new Session(new SessionOptions { Preset = "chrome-latest" });

var formData = new Dictionary<string, string> {
    { "user", "alice" },
    { "token", "abc 123" }
};
var r = s.PostForm("https://httpbin.org/post", formData);
Console.WriteLine(r.Text);

Things to keep in mind:

Encoding is fixed. Always UTF-8 percent-encoded. Don't try Latin-1 in form bodies even if a server "would accept it."
Repeated keys. url-encoded supports the same key twice (a=1&a=2). Most builders give you a list-valued helper. In Go: form.Add("a", "1"); form.Add("a", "2").
Length limits. Some servers cap form bodies around 1MB. If you're shipping more than a kilobyte of text, multipart is usually the better fit. Megabytes? You almost certainly want multipart for file fields, or just JSON.

Multipart

Multipart wraps each field in its own MIME part with a boundary string. You get a Content-Type like multipart/form-data; boundary=----WebKitFormBoundaryAbCdEf123. The body looks roughly like:

------WebKitFormBoundaryAbCdEf123
Content-Disposition: form-data; name="comment"

hello there
------WebKitFormBoundaryAbCdEf123
Content-Disposition: form-data; name="file"; filename="note.txt"
Content-Type: text/plain

file body bytes
------WebKitFormBoundaryAbCdEf123--

You almost never want to write this by hand. Use the helpers, that's what they're for.

Go
Python
Node.js
.NET

httpcloak ships MultipartField and BuildMultipart for the common case.

package main

import (
    "bytes"
    "context"
    "fmt"

    httpcloak "github.com/sardanioss/httpcloak"
)

func main() {
    s := httpcloak.NewSession("chrome-latest")
    defer s.Close()

    fields := []httpcloak.MultipartField{
        {Name: "comment", Value: "hello there"},
        {
            Name:        "file",
            Filename:    "note.txt",
            Content:     []byte("file body bytes"),
            ContentType: "text/plain",
        },
    }
    body, contentType, err := httpcloak.BuildMultipart(fields)
    if err != nil {
        panic(err)
    }

    req := &httpcloak.Request{
        Method: "POST",
        URL:    "https://httpbin.org/post",
        Headers: map[string][]string{
            "Content-Type": {contentType},
        },
        Body: bytes.NewReader(body),
    }
    resp, _ := s.Do(context.Background(), req)
    defer resp.Close()

    body2, _ := resp.Text()
    fmt.Println(body2)
    // form: { "comment": "hello there" }
    // files: { "file": "file body bytes" }
}

BuildMultipart handles boundary generation, MIME headers per part, and the trailing --. The Content-Type it returns is the full string with the boundary parameter, drop it straight into your headers.

For really large file uploads, building the whole body in memory is a bad idea. Use mime/multipart.Writer directly with io.Pipe and stream into the request via Body: pipeReader. Your RAM will thank you.

r = s.post(
    "https://httpbin.org/post",
    data={"comment": "hello there"},
    files={"file": ("note.txt", b"file body bytes", "text/plain")},
)

result = r.json()
print(result["form"])   # {"comment": "hello there"}
print(result["files"])  # {"file": "file body bytes"}

The files= kwarg accepts:

{"name": file_object}: open file, the binding pulls filename and bytes.
{"name": (filename, bytes_or_str)}: explicit filename, content-type sniffed.
{"name": (filename, bytes_or_str, content_type)}: full control.

const r = await s.post("https://httpbin.org/post", {
  form: { comment: "hello there" },
  files: {
    file: {
      filename: "note.txt",
      content: Buffer.from("file body bytes"),
      contentType: "text/plain",
    },
  },
});

console.log(r.json().form);   // { comment: "hello there" }
console.log(r.json().files);  // { file: "file body bytes" }

using HttpCloak;

using var s = new Session(new SessionOptions { Preset = "chrome-latest" });

var fields = new Dictionary<string, string> {
    { "comment", "hello there" }
};
var files = new Dictionary<string, MultipartFile> {
    { "file", new MultipartFile {
        Filename = "note.txt",
        Content = System.Text.Encoding.UTF8.GetBytes("file body bytes"),
        ContentType = "text/plain"
    }}
};

var r = s.PostMultipart("https://httpbin.org/post", fields: fields, files: files);
Console.WriteLine(r.Text);

Order of fields

Multipart preserves the order you list fields. For most servers that doesn't matter (they parse into a map). But occasionally:

A server rejects an upload if the file part comes before a CSRF token field.
A server expects a specific field order baked into its form validation.

Seeing weird "field missing" or "invalid form" errors when the values look fine? Try reordering. Drop the file last, drop the token first. The browser-emitted order is whatever the <form> markup says, which is usually text fields then file inputs.

Boundary string

Browsers use boundaries like ----WebKitFormBoundaryAbCdEf123456. Go's stdlib (and therefore httpcloak's BuildMultipart) generates random boundaries that look different. For 99% of servers this doesn't matter, the boundary is just a delimiter. If you hit a strict WAF that pattern-matches on browser-style boundaries, you can build the body yourself with mime/multipart.Writer.SetBoundary() and pass a Chrome-shaped one. Rare scenario though.

Verify with httpbin

POST https://httpbin.org/post is the one-stop shop for verifying multipart serialization. The response gives you:

form: text fields
files: file fields, with the file content inlined as a string
headers["Content-Type"]: the Content-Type you sent (verify the boundary)

If files is empty but you sent a file, your boundary or part headers are busted. If the file content looks like garbage, your encoding's off.

URL-encoded​

Multipart​

Order of fields​

Boundary string​

Verify with httpbin​

URL-encoded

Multipart

Order of fields

Boundary string

Verify with httpbin