如何高效地从 REST API 下载文件并将其发送到消息代理(例如 RabbitMQ)?

huangapple 未分类评论45阅读模式
英文:

how to efficiently download a file from rest api and send it to a message brocker like RabbitMq?

问题

目标是从URI下载文件并将其映射到现有数据模型。这个数据模型称为FileNode,并且具有一个属性,其类型为InputStream(此InputStream表示文件数据)。我不知道如何做到这一点。我尝试使用Spring的RestTemplate,但它在其doExecute方法中关闭了响应的InputStream(所有RestTemplate模板方法最终都委托给此方法)。然后我尝试扩展RestTemplate并注释掉关闭响应InputStream的代码:

@Override
protected <T> T doExecute(URI url, HttpMethod method, RequestCallback requestCallback, ResponseExtractor<T> responseExtractor) throws RestClientException {
  Assert.notNull(url, "URI is required");
  Assert.notNull(method, "HttpMethod is required");
  ClientHttpResponse response = null;
  try {
    ClientHttpRequest request = createRequest(url, method);
    if (requestCallback != null) {
      requestCallback.doWithRequest(request);
    }
    response = request.execute();
    handleResponse(url, method, response);
    return (responseExtractor != null ? responseExtractor.extractData(response) : null);
  }
  catch (IOException ex) {
    String resource = url.toString();
    String query = url.getRawQuery();
    resource = (query != null ? resource.substring(0, resource.indexOf('?')) : resource);
    throw new ResourceAccessException("I/O error on " + method.name() +
      " request for \"" + resource + "\": " + ex.getMessage(), ex);
  }
  finally {
    /**** when to close the response? can I use a proxy for the response's InputStream that closes this response when itself is closed?****/
    if (!(responseExtractor instanceof CloseableResponseExtractor)) //?!
      if (response != null) {
        response.close();
      }
  }
}

然后我在考虑应该有更简单的方法来做这件事,所以我发布了这个问题。

**注意:**我立即将文件发送到消息代理,我的意思是我不必将其保存在内存中。我认为从内存或磁盘保存文件,然后从那里进行流式处理不是一个好主意,因为文件大小可能太大,我不知道!实际上,我的代码必须是基础架构(遗留代码+消息代理+...)和REST API之间的非常薄的层,我认为如果我首先将其写入磁盘,它会严重影响响应时间(我需要从磁盘写入和读取!)。你有什么建议?

在这种情况下,最佳做法是什么,通常是如何处理的?

谢谢!

英文:

the objective is to download a file from a URI and map it to an existing data model. this data model called FileNode and has a property with type InputStream (this InputStream represents the file data). I do not know how to do this. I tried to use Spring RestTemplate, but it closes the response InputStream in its doExecute method (all RestTemplate template methods eventually delegates to this method). then I tried to extend RestTemplate and comment the code that closes the Response InputStream :

  @Override
  protected &lt;T&gt; T doExecute(URI url, HttpMethod method, RequestCallback requestCallback, ResponseExtractor&lt;T&gt; responseExtractor) throws RestClientException {
    Assert.notNull(url, &quot;URI is required&quot;);
    Assert.notNull(method, &quot;HttpMethod is required&quot;);
    ClientHttpResponse response = null;
    try {
      ClientHttpRequest request = createRequest(url, method);
      if (requestCallback != null) {
        requestCallback.doWithRequest(request);
      }
      response = request.execute();
      handleResponse(url, method, response);
      return (responseExtractor != null ? responseExtractor.extractData(response) : null);
    }
    catch (IOException ex) {
      String resource = url.toString();
      String query = url.getRawQuery();
      resource = (query != null ? resource.substring(0, resource.indexOf(&#39;?&#39;)) : resource);
      throw new ResourceAccessException(&quot;I/O error on &quot; + method.name() +
        &quot; request for \&quot;&quot; + resource + &quot;\&quot;: &quot; + ex.getMessage(), ex);
    }
    finally {
      /**** when to close the response? can I use a proxy for the response&#39;s InputStream that closes this response when itself is closed?****/
      if(!(responseExtractor instanceof CloseableResponseExtractor)) //?!
      if (response != null) {
        response.close();
      }
    }
  }

then I was thinking it should be a simpler way to do this, so I posted this question.

note: I send the file immediately to a message broker, I mean I do not have to save it in memory. and I do not think it is good idea to save the file in memory or disk and then stream from there because the file size might be too large, I do not know! in fact, my code must be a very thin layer between the infrastructure (legacy codes + message broker+...) and the rest API, and I think if I first write it to disk it impacts the response time severely (I need to write and read from disk!). what are your recommendations?

what is the best practice for this scenario and usually how this is done?

thank you!

huangapple
  • 本文由 发表于 2020年4月9日 20:44:39
  • 转载请务必保留本文链接:https://java.coder-hub.com/61121464.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定