Skip to main content

Data.fetch_many

Data.fetch_manyurismethodheadersformatreturntime_between_requestscache_policyon_problems

Group: Web
Aliases: download, http get

Documentation​

Read data from a set of URIs.

Fetches from the provided URI and returns the response, parsing the body if the content-type is recognised. Returns an error if the status code does not represent a successful response.

Arguments​

  • uris: The set of URI to fetch.
  • method: The HTTP method to use. Must be one of HTTP_Method.Get, HTTP_Method.Head, HTTP_Method.Delete, HTTP_Method.Options. Defaults to HTTP_Method.Get.
  • headers: The headers to send with the request. Defaults to an empty vector.
  • format: The format to use for interpreting the response. Defaults to Auto_Detect. If Raw_Response is selected or if the format cannot be determined automatically, a raw HTTP Response will be returned.
  • return: Specifies the shape of the data to return.
  • time_between_requests: The time to wait between requests, in milliseconds. Defaults to 0 (no wait).
  • cache_policy: The caching policy to use. Defaults to Default. See Request Caching for details.
  • on_problems: Specifies the behavior when a problem occurs during the function. By default, if one of the uris fails to fetch, a warning is issued and the entry for that uri becomes Nothing, but the operation proceeds. If set to Report_Error, the operation fails with a dataflow error on the first failing file. If set to Ignore, the operation proceeds without errors or warnings, replacing files that fail to load with Nothing.

Remarks​

Request Caching​

Responses to HTTP data requests are cached, and additional requests for the same resources will use the cache, saving a round-trip call to the remote server. Two resources are considered the same if the URIs and request headers are the same. Header order does not affect sameness. The cache respects the "max-age" and "Age" response headers; see Data.fetch for more details. The cached values are retained as long as the project remains open. Closing a project will clear the cache.