Skip to main content

Data.fetch

Data.fetchurimethodheadersformat cache_policy

Group: Input
Aliases: download, http get

Documentation​

Fetches from the provided URI and returns the response, parsing the body if the content-type is recognised. Returns an error if the status code does not represent a successful response.

Arguments​

  • uri: The URI to fetch.
  • method: The HTTP method to use. Must be one of HTTP_Method.Get, HTTP_Method.Head, HTTP_Method.Delete, HTTP_Method.Options. Defaults to HTTP_Method.Get.
  • headers: The headers to send with the request. Defaults to an empty vector.
  • format: The format to use for interpreting the response. Defaults to Auto_Detect. If Raw_Response is selected or if the format cannot be determined automatically, a raw HTTP Response will be returned.

Examples​

Read from an HTTP endpoint.​

      import Standard.Base.Data
      response = Data.fetch URL

Read from an HTTP endpoint and write the results to a file.​

      import Standard.Base.Data
      file = enso_project.data / "spreadsheet.xls"
      Data.fetch URL . body . write file

Remarks​

Request Caching​

Responses to HTTP data requests are cached, and additional requests for the same resources will use the cache, saving a round-trip call to the remote server. Two resources are considered the same if the URIs and request headers are the same. Header order does not affect sameness.

The cached values are retained as long as the project remains open. Closing a project will clear the cache.

The cache respects the "max-age" and "Age" response headers received from remote servers. These headers are used to determine if the cached value is fresh or stale. If it is stale, the cached value is removed and a request is made again to the remote servers.

The following limits are imposed on values stored in the cache:

  • Single file limit: a single file can be no more than 10M.
  • Total cache size limit: the entire cache can be no more than 10G.

For data responses over the single file limit, you can use Data.download to download the file locally. Download sizes are not constrained by either limit.

If the entire cache goes over the total cache size limit, the least-recently-used entries are removed.