詳解HDI Implementation中的預(yù)覽流程

作者：潤和軟件 郭新星

相機(jī)作為智能手機(jī)上少有的成長空間不錯的，能夠做出差異化的功能，每年都能成為各大Android手機(jī)廠商爭相宣傳的亮點(diǎn)。眾所周知Android采用Linux 作為其內(nèi)核，而Linux采用的開源協(xié)議具有傳染性[1]，導(dǎo)致Android HAL[2]成為了手機(jī)廠商們競爭的重要戰(zhàn)場。隨著OpenHarmony 3.1[3]的發(fā)布，相機(jī)模塊也逐漸完善起來，目前提供了基礎(chǔ)預(yù)覽和拍照的能力。OpenHarmony中，相機(jī)用戶態(tài)驅(qū)動框架承擔(dān)了和Android Camera HAL一樣的角色，這部分位于OpenHarmony的HDF[4]中，對上實(shí)現(xiàn)相機(jī)HDI[5]接口，對下實(shí)現(xiàn)相機(jī)Pipeline模型，管理相機(jī)各個硬件設(shè)備。

相機(jī)用戶態(tài)驅(qū)動框架（下圖的CameraHost 部分）總體可以分為三層，HDI實(shí)現(xiàn)層，實(shí)現(xiàn)相機(jī)標(biāo)準(zhǔn)南向接口；框架層，對接HDI實(shí)現(xiàn)層的控制、流的轉(zhuǎn)發(fā)，實(shí)現(xiàn)數(shù)據(jù)通路的搭建、管理相機(jī)各個硬件設(shè)備等功能；適配層，屏蔽底層芯片和OS差異，支持多平臺適配。

模塊介紹
HDI Implementation：對上實(shí)現(xiàn)HDI接口，向下調(diào)用框架層的接口，完成HDI接口任務(wù)的轉(zhuǎn)發(fā)。

Buffer Manager ：屏蔽不同內(nèi)存管理的差異，為子系統(tǒng)提供統(tǒng)一的操作接口，同時提供buffer輪轉(zhuǎn)的功能。

Pipeline Core ：解析HCS配置完成pipeline的搭建，調(diào)度pipeline中的各個node完成流的處理
Device Manager：通過調(diào)用底層硬件適配層接口，實(shí)現(xiàn)查詢控制底層設(shè)備、枚舉監(jiān)聽底層設(shè)備的功能。

Platform Adaption ：屏蔽硬件差異，為Device Manager提供統(tǒng)一的操作底層硬件的能力。

目錄結(jié)構(gòu)

Shelldrivers/peripheral/camera|-- README_zh.md|-- bundle.json|-- figures|  `-- logic-view-of-modules-related-to-this-repository_zh.png|-- hal|  |-- BUILD.gn|  |-- adapter|  |-- buffer_manager|  |-- camera.gni|  |-- device_manager|  |-- hdi_impl|  |-- include|  |-- init|  |-- pipeline_core|  |-- test|  `-- utils|-- hal_c|  |-- BUILD.gn|  |-- camera.gni|  |-- hdi_cif|  `-- include`-- inteRFaces  |-- hdi_ipc  |-- hdi_passthrough  `-- include

（左右移動查看全部內(nèi)容）

HDI Implementation中的預(yù)覽流程

接下來我們通過已經(jīng)發(fā)布的OpenHarmony 3.1開源代碼，來看看預(yù)覽是怎么完成的吧

drivers/peripheral/camera/hal/test/v4l2/src /preview_test.cpp存放了針對v4l2的預(yù)覽測試代碼，入口如下：

C++TEST_F(UtestPreviewTest, camera_preview_0001){  std::cout << "==========[test log] Preview stream, expected success." << std::endl;  // Get the stream manager  display_->AchieveStreamOperator(); // 獲取stream operator  // start stream  display_->intents = {Camera::PREVIEW}; // 預(yù)覽流  display_->StartStream(display_->intents); // 起流  // Get preview  display_->StartCapture(display_->streamId_preview, display_->captureId_preview, false, true);  // release stream  display_->captureIds = {display_->captureId_preview};  display_->streamIds = {display_->streamId_preview};  display_->StopStream(display_->captureIds, display_->streamIds);}

（左右移動查看全部內(nèi)容）

先獲取stream operator實(shí)例

C++void testdisplay::achievestreamoperator(){  // create and get streamoperator information  std::shared_ptr streamoperatorcallback =    std::make_shared();  rc = cameradevice->getstreamoperator(streamoperatorcallback, streamoperator);     // ........}

（左右移動查看全部內(nèi)容）

通過前文的streamOperator創(chuàng)建流

C++void TestDisplay::StartStream(std::vector intents){  // ..............................  for (auto& intent : intents) {    if (intent == 0) {      std::shared_ptr producer = IBufferProducer::CreateBufferQueue();      producer->SetQueueSize(8); // 創(chuàng)建buffer的生產(chǎn)端，并和相應(yīng)的流進(jìn)行綁定      auto callback = [this](std::shared_ptr Prebuffer) {        BufferCallback(Prebuffer, preview_mode);        return;      };      producer->SetCallback(callback);      streamInfo->streamId_ = streamId_preview;      streamInfo->width_ = 640; // 640:picture width      streamInfo->height_ = 480; // 480:picture height      streamInfo->format_ = CAMERA_FORMAT_YUYV_422_PKG;      streamInfo->datasapce_ = 8; // 8:picture datasapce      streamInfo->intent_ = intent;      streamInfo->tunneledMode_ = 5; // 5:tunnel mode      streamInfo->bufferQueue_ = producer;      streamInfos.push_back(streamInfo);    } else if (intent == 1) {   // .......................  }  rc = streamOperator->CreateStreams(streamInfos); // 創(chuàng)建流  // ................................  rc = streamOperator->CommitStreams(Camera::NORMAL, ability); // 提交流  // .................................}

（左右移動查看全部內(nèi)容）

下面我們正式進(jìn)入到hal的源代碼中看看是怎么創(chuàng)建流的吧

C++CamRetCode StreamOperator::CreateStreams(const std::vector<std::shared_ptr>& streamInfos){ // .....  for (auto it : streamInfos) {//....    std::shared_ptr stream = StreamFactory::Instance().CreateShared(      IStream::g_availableStreamType[it->intent_], it->streamId_, it->intent_, pipelineCore_, messenger_); // 創(chuàng)建流實(shí)例// ...    StreamConfiguration scg;    scg.id = it->streamId_;    scg.type = it->intent_;    scg.width = it->width_;    scg.height = it->height_;    PixelFormat pf = static_cast(it->format_);    scg.format = BufferAdapter::PixelFormatToCameraFormat(pf);    scg.dataspace = it->datasapce_;    scg.tunnelMode = it->tunneledMode_;    scg.minFrameDuration = it->minFrameDuration_;    scg.encodeType = it->encodeType_;
    RetCode rc = stream->ConfigStream(scg); // 依據(jù)上文的流信息配置流// ...    if (it->bufferQueue_ != nullptr) { // 綁定前文的生產(chǎn)端      auto tunnel = std::make_shared();      CHECK_IF_PTR_NULL_RETURN_VALUE(tunnel, INSUFFICIENT_RESOURCES);      RetCode rc = tunnel->AttachBufferQueue(it->bufferQueue_);      CHECK_IF_NOT_EQUAL_RETURN_VALUE(rc, RC_OK, INVALID_ARGUMENT);      if (stream->AttachStreamTunnel(tunnel) != RC_OK) {        CAMERA_LOGE("attach buffer queue to stream [id = %{public}d] failed", it->streamId_);        return INVALID_ARGUMENT;      }    }    {      std::lock_guard<std::mutex> l(streamLock_);      streamMap_[stream->GetStreamId()] = stream; // 保存流實(shí)例    }// ...}

（左右移動查看全部內(nèi)容）

從上面可以看出，消費(fèi)端傳遞到了hal，那必然是由hal從bufferproducer獲取buffer，并觸發(fā)預(yù)覽的啟動流程。那看看AttachStreamTunnel 的實(shí)現(xiàn)吧

C++RetCode StreamBase::AttachStreamTunnel(std::shared_ptr& tunnel){  if (state_ == STREAM_STATE_BUSY || state_ == STREAM_STATE_OFFLINE) {    return RC_ERROR;  }
  tunnel_ = tunnel; // 綁定生產(chǎn)端  CHECK_IF_PTR_NULL_RETURN_VALUE(tunnel_, RC_ERROR);  tunnel_->SetBufferCount(GetBufferCount()); // 配置輪轉(zhuǎn)的buffer個數(shù)  TunnelConfig config = {(uint32_t)streamConfig_.width, (uint32_t)streamConfig_.height,    (uint32_t)streamConfig_.format, streamConfig_.usage};  tunnel_->Config(config);
  streamConfig_.tunnelMode = true;  return RC_OK;}

（左右移動查看全部內(nèi)容）

CreateStream之后便是CommitStream，這里的CommitStream 做了些什么事情呢，我們接著往下看

C++CamRetCode StreamOperator::CommitStreams(OperationMode mode,                     const std::shared_ptr& modeSetting){// ......  std::vector configs = {};  {    std::lock_guard<std::mutex> l(streamLock_);    for (auto it : streamMap_) { // 獲取流的配置，前文CreateStrea時保存的流      configs.emplace_back(it.second->GetStreamAttribute());    }  } // 檢查流是否被支持  DynamicStreamSwitchMode method = streamPipeline_->CheckStreamsSupported(mode, modeSetting, configs);  if (method == DYNAMIC_STREAM_SWITCH_NOT_SUPPORT) {    return INVALID_ARGUMENT;  }  if (method == DYNAMIC_STREAM_SWITCH_NEED_INNER_RESTART) {    std::lock_guard<std::mutex> l(streamLock_);    for (auto it : streamMap_) {      it.second->StopStream();// 如果流被支持，但需要內(nèi)部重啟，這里先停流    }  }  {    std::lock_guard<std::mutex> l(streamLock_);    for (auto it : streamMap_) {      if (it.second->CommitStream() != RC_OK) { // 真正的 CommitStream，下面再細(xì)說        CAMERA_LOGE("commit stream [id = %{public}d] failed.", it.first);        return DEVICE_ERROR;      }    }  }  RetCode rc = streamPipeline_->PreConfig(modeSetting); // 把模式傳入進(jìn)行預(yù)配置  if (rc != RC_OK) {    CAMERA_LOGE("prepare mode settings failed");    return DEVICE_ERROR;  }  rc = streamPipeline_->CreatePipeline(mode);// 創(chuàng)建pipeline  if (rc != RC_OK) {    CAMERA_LOGE("create pipeline failed.");    return INVALID_ARGUMENT;  }
  DFX_LOCAL_HITRACE_END;  return NO_ERROR;}

（左右移動查看全部內(nèi)容）

C++RetCode StreamBase::CommitStream(){// ...  hostStreamMgr_ = pipelineCore_->GetHostStreamMgr(); //從pipelinecore獲取hoststreamanager  CHECK_IF_PTR_NULL_RETURN_VALUE(hostStreamMgr_, RC_ERROR);// ...    info.bufferPoolId_ = poolId_;    info.bufferCount_ = GetBufferCount();  // 初始化 bufferpool    RetCode rc = bufferPool_->Init(streamConfig_.width, streamConfig_.height, streamConfig_.usage,                    streamConfig_.format, GetBufferCount(), CAMERA_BUFFER_SOURCE_TYPE_EXTERNAL);    if (rc != RC_OK) {      CAMERA_LOGE("stream [id:%{public}d] initialize buffer pool failed.", streamId_);      return RC_ERROR;    }  }// stream傳遞到pipelinecore 并進(jìn)行綁定  RetCode rc = hostStreamMgr_->CreateHostStream(info, [this](std::shared_ptr buffer) {    HandleResult(buffer);    return;  });// ....  return RC_OK;}

（左右移動查看全部內(nèi)容）

CreateStream 和CommitStream結(jié)束之后便是Capture，這里包含了起流的動作，關(guān)鍵實(shí)現(xiàn)如下

C++CamRetCode StreamOperator::Capture(int captureId, const std::shared_ptr& captureInfo, bool isStreaming){// ...//  captureId 捕獲請求的id; captureInfo 預(yù)覽/拍照/錄像的參數(shù)；isStreaming 連續(xù)捕獲還是單次捕獲（拍照）  CaptureSetting setting = captureInfo->captureSetting_;  auto request =    std::make_shared(captureId, captureInfo->streamIds_.size(), setting,                     captureInfo->enableShutterCallback_, isStreaming);  for (auto id : captureInfo->streamIds_) {    // 創(chuàng)建捕獲請求，并傳遞給前文創(chuàng)建的流    RetCode rc = streamMap_[id]->AddRequest(request);    if (rc != RC_OK) {      return DEVICE_ERROR;    }  }// ...}

（左右移動查看全部內(nèi)容）

從上面的代碼可知預(yù)覽、拍照、錄像都是通過捕獲請求觸發(fā)，單次拍照則為單次捕獲請求，預(yù)覽和錄像則是連續(xù)捕獲請求。

C++RetCode StreamBase::AddRequest(std::shared_ptr& request){  CHECK_IF_PTR_NULL_RETURN_VALUE(request, RC_ERROR);  request->AddOwner(shared_from_this());
  request->SetFirstRequest(false);  if (isFirstRequest) {    RetCode rc = StartStream(); // 起流    if (rc != RC_OK) {      CAMERA_LOGE("start stream [id:%{public}d] failed", streamId_);      return RC_ERROR;    }    request->SetFirstRequest(true);    isFirstRequest = false;  }  {    std::unique_lock<std::mutex> l(wtLock_);    waitingList_.emplace_back(request); // 捕獲請求添加到waitingList    cv_.notify_one();  }  return RC_OK;}

（左右移動查看全部內(nèi)容）

看看StreamStream是怎么實(shí)現(xiàn)的吧

C++RetCode StreamBase::StartStream(){// ...  RetCode rc = pipeline_->Prepare({streamId_}); // pipeline先完成一些準(zhǔn)備工作// ...
  state_ = STREAM_STATE_BUSY;  std::string threadName =    g_availableStreamType[static_cast(streamType_)] + "#" + std::to_string(streamId_);  handler_ = std::make_unique<std::thread>([this, &threadName] {// 創(chuàng)建輪轉(zhuǎn)線程    prctl(PR_SET_NAME, threadName.c_str());    while (state_ == STREAM_STATE_BUSY) {      HandleRequest(); // 處理捕獲請求    }  });// ...  rc = pipeline_->Start({streamId_}); // 通知pipeline和底層硬件可以開始出幀了// ...  return RC_OK;}

（左右移動查看全部內(nèi)容）

C++void StreamBase::HandleRequest(){  // 如果有 捕獲請求下發(fā)，則退出等待狀態(tài)  if (waitingList_.empty()) {    std::unique_lock<std::mutex> l(wtLock_);    if (waitingList_.empty()) {      cv_.wait(l, [this] { return !(state_ == STREAM_STATE_BUSY && waitingList_.empty()); });    }  }// ...    request = waitingList_.front();    CHECK_IF_PTR_NULL_RETURN_VOID(request);    if (!request->IsContinous()) { // 如果是連續(xù)捕獲，則保留一份拷貝在waitinglist      waitingList_.pop_front();    }  }// 處理捕獲請求  request->Process(streamId_);// 最終調(diào)用下面的Capture接口  return;}

（左右移動查看全部內(nèi)容）

C++RetCode StreamBase::Capture(const std::shared_ptr& request){  CHECK_IF_PTR_NULL_RETURN_VALUE(request, RC_ERROR);  CHECK_IF_PTR_NULL_RETURN_VALUE(pipeline_, RC_ERROR);
  RetCode rc = RC_ERROR;  if (request->IsFirstOne() && !request->IsContinous()) {    uint32_t n = GetBufferCount();    for (uint32_t i = 0; i < n; i++) {      DeliverBuffer();// 單次捕獲一次性下發(fā)所有的buffer    }  } else {    do {      rc = DeliverBuffer();// 連續(xù)捕獲每次下發(fā)一個buffer    } while (rc != RC_OK && state_ == STREAM_STATE_BUSY);  }
  if (request->NeedCancel()) {// 被取消的捕獲則退出    CAMERA_LOGE("StreamBase::Capture stream [id:%{public}d] request->NeedCancel", streamId_);    return RC_OK;  }
  rc = pipeline_->Config({streamId_}, request->GetCaptureSetting());// 通知pipeline配置  if (rc != RC_OK) {    CAMERA_LOGE("stream [id:%{public}d] config pipeline failed.", streamId_);    return RC_ERROR;  }
  rc = pipeline_->Capture({streamId_}, request->GetCaptureId());// 這里的capture指的是pipeline中的source node開始回buffer
  {    std::unique_lock<std::mutex> l(tsLock_);    inTransitList_.emplace_back(request);// 處理過的捕獲請求存放在inTransitList  }  return RC_OK;}

（左右移動查看全部內(nèi)容）

到這起流的流程就結(jié)束了，pipeline回上來的幀通過OnFrame接口處理

C++RetCode StreamBase::OnFrame(const std::shared_ptr& request){// ...  bool isEnded = false;  if (!request->IsContinous()) {    isEnded = true;  } else if (request->NeedCancel()) {    isEnded = true;  }  {    // inTransitList_ may has multiple copies of continious-capture request, we just need erase one of them.    std::unique_lock<std::mutex> l(tsLock_);    for (auto it = inTransitList_.begin(); it != inTransitList_.end(); it++) {      if ((*it) == request) {        inTransitList_.erase(it);// 已經(jīng)回幀的請求，從inTransitList刪除        break;      }    }    if (isEnded) {      // if this is the last request of capture, send CaptureEndedMessage.      auto it = std::find(inTransitList_.begin(), inTransitList_.end(), request);      if (it == inTransitList_.end()) {        std::shared_ptr endMessage =          std::make_shared(streamId_, request->GetCaptureId(), request->GetEndTime(),                             request->GetOwnerCount(), tunnel_->GetFrameCount());        CAMERA_LOGV("end of stream [%d], ready to send end message, capture id = %d",          streamId_, request->GetCaptureId());        messenger_->SendMessage(endMessage);        pipeline_->CancelCapture({streamId_});// 如果此次捕獲結(jié)束，則取消捕獲      }    }  }  ReceiveBuffer(buffer);// 底層返回的buffer送還到生產(chǎn)端，最終幀數(shù)據(jù)送到消費(fèi)端  return RC_OK;}

（左右移動查看全部內(nèi)容）

附錄：

• linux和Android的關(guān)系 - 知乎 (zhihu.com)

• HAL Subsystem | Android Open Source Project (google.cn)

• zh-cn/release-notes/OpenHarmony-v3.1-release.md · OpenHarmony/docs - Gitee.com

• OpenHarmony HDF 驅(qū)動框架介紹和驅(qū)動加載過程分析-OpenHarmony技術(shù)社區(qū)

• OpenHarmony HDF HDI基礎(chǔ)能力分析與使用