CANN/hcomm源代码构建指南

Source Code Build【免费下载链接】hcommHCOMMHuawei Communication是HCCL的通信基础库提供通信域以及通信资源的管理能力。项目地址: https://gitcode.com/cann/hcommEnvironment PreparationThis project supports source code building. Before compiling and running, complete the basic environment setup and source code download by following the steps below. Ensure that the NPU driver, firmware, and CANN software are installed.PrerequisitesThe software dependencies required for compiling this project are listed below. Ensure that the version requirements are met.python 3.7.0pip3 20.3.0setuptools 45.0.0wheel 0.34.0gcc and g: 7.3.0 to 13.3.xcmake 3.16.0pkg-config 0.29.1 (for compiling rdma-core)patch 2.7.0 (for applying patch files)ccache (optional, for improving incremental compilation speed)lcov (optional, for generating UT or ST coverage reports)Installing the CANN Software PackageInstall the driver and firmware (runtime dependency)For downloading and installing the driver and firmware, refer to the Prepare Software Package and Install NPU Driver and Firmware sections in the CANN Software Installation Guide. The driver and firmware are runtime dependencies. If you are only compiling the source code of this project, they do not need to be installed.Install the CANN software packageScenario 1: Experience or develop based on the master versionClick the download link, select the latest version, and download the corresponding package based on the product model and environment architecture. The installation commands are as follows. For more guidance, refer to the CANN Software Installation Guide.Install the CANN Toolkit development kit package.# Ensure the installation package has executable permissions chmod x Ascend-cann-toolkit_${cann_version}_linux-${arch}.run # Installation command ./Ascend-cann-toolkit_${cann_version}_linux-${arch}.run --install --install-path${install_path}Install the CANN ops operator package (runtime dependency).The ops operator package is a runtime dependency. If you are only compiling the source code of this project, this package does not need to be installed.# Ensure the installation package has executable permissions chmod x Ascend-cann-${soc_name}-ops_${cann_version}_linux-${arch}.run # Installation command ./Ascend-cann-${soc_name}-ops_${cann_version}_linux-${arch}.run --install --install-path${install_path}${cann_version}: Indicates the CANN software package version number.${arch}$: Indicates the CPU architecture, for example, aarch64 or x86_64.${soc_name}: Indicates the NPU model name.${install_path}: Indicates the specified installation path. The CANN ops operator package must be installed in the same path as the CANN Toolkit development kit package. The default installation path for the root user is/usr/local/Ascend.Scenario 2: Experience or develop based on a released versionVisit the CANN official download center, select a released version (only CANN 8.5.0 and later versions are supported), and download the corresponding package based on the product model and environment architecture. Finally, follow the commands provided on the webpage to complete the installation.Environment VerificationAfter installing the CANN software package, verify that the environment is functioning correctly.Check the NPU device:# Run npu-smi. If device information is displayed normally, the driver is working correctly. npu-smi infoCheck the CANN software:# View the version information provided by the version field of the CANN Toolkit development kit package (default installation path). arch indicates the CPU architecture (aarch64 or x86_64). cat /usr/local/Ascend/cann/arch-linux/ascend_toolkit_install.info # View the version information provided by the version field of the CANN ops operator package (default installation path). cat /usr/local/Ascend/cann/arch-linux/ascend_ops_install.infoEnvironment Variable ConfigurationSelect the appropriate command to apply the environment variables.# Default installation path, using the root user as an example (for non-root users, replace /usr/local with ${HOME}) source /usr/local/Ascend/cann/set_env.sh # Specified installation path # source ${install_path}/cann/set_env.shCompilation and InstallationDownloading the Source CodeUse the following command to download the source code. Replace ${tag_version} with the target branch tag name. For the mapping between source branch tags and CANN versions, refer to the release repository.# Download the source code for the corresponding project branch git clone -b ${tag_version} https://gitcode.com/cann/hcomm.gitCompiling the Source CodeThis project provides a one-click build capability. Navigate to the root directory of the repository and execute the following commands:# Compile the host package bash build.sh --pkg # Compile the host and device package bash build.sh --pkg --fullDuring compilation, the dependency packages listed in Open Source Third-Party Software Dependencies are automatically downloaded. If the compilation environment does not have network access, download the required dependency packages in a networked environment, manually upload them to the compilation environment, and specify the dependency package storage path using the--cann_3rd_lib_pathparameter.# Specify the dependency package storage path. Default: ./third_party bash build.sh --cann_3rd_lib_path{your_3rd_party_path}After compilation, acann-hcomm_version_linux-arch.runsoftware package is generated in the./build_outdirectory.versionindicates the software version number, andarchindicates the operating system architecture. The values include x86_64 and aarch64.InstallationInstall the compiled HCOMM software package:bash ./build_out/cann-hcomm_version_linux-arch.run --fullDuring installation, replace the package name in the command with the actual software package name.After installation, the HCOMM software package generated by the user replaces the HCOMM-related software in the installed CANN Toolkit development kit package.UninstallationTo uninstall the compiled HCOMM software package and restore it to the state after installing the CANN Toolkit development kit package, use the following command:bash ./build_out/cann-hcomm_version_linux-arch.run --uninstallDuring uninstallation, replace the package name in the command with the actual software package name.TestingLLT TestingAfter installing the compiled HCOMM software package, execute the following command to run LLT test cases:bash build.sh --utOn-Board TestingNoteBefore on-board testing, ensure that the driver, firmware, CANN Toolkit development kit package, and CANN ops operator package are installed.Developers can use the HCCL Test tool for collective communication function and performance testing on the board. The workflow for using the HCCL Test tool is as follows:Tool compilationBefore using the HCCL Test tool, install the MPI dependency and compile the HCCL Test tool. For detailed instructions, refer to the MPI Installation and Configuration and Tool Compilation sections in the corresponding version of the Ascend Documentation Center - HCCL Performance Test Tool Guide.Disable signature verificationThecann-hcomm_version_linux-arch.runsoftware package generated from this source repository contains the following tar.gz subpackages:cann-hcomm-compat.tar.gz: HCOMM compatibility upgrade package.cann-hccd-compat.tar.gz: DataFlow compatibility upgrade package.aicpu_hcomm.tar.gz: AI CPU communication base package.These tar.gz packages are loaded to the Device when the service starts. During the loading process, the driver performs security signature verification by default to ensure the package is trusted. Because the tar.gz packages compiled from this source repository do not contain a signature header, the driver security signature verification mechanism needs to be disabled.Method to disable signature verification:Use Ascend HDK 25.5.T2.B001 or later, and use the npu-smi tool provided with the Ascend HDK to disable signature verification. The following are reference commands. Execute them as the root user on the physical machine (using device 0 as an example).npu-smi set -t custom-op-secverify-enable -i 0 -d 1 # Enable signature verification configuration npu-smi set -t custom-op-secverify-mode -i 0 -d 0 # Disable custom signature verificationExecute the HCCL Test command to test the function and performance of collective communication.Using one compute node, 8 NPU devices, and testing the AllReduce operator performance as an example:# /usr/local/Ascend is the CANN software installation path for the root user under the default installation path. Replace it with the actual path. cd /usr/local/Ascend/ascend-toolkit/latest/tools/hccl_test # Data size (-b) from 8KB to 64MB, increment factor (-f) of 2, number of NPUs participating in training is 8 mpirun -n 8 ./bin/all_reduce_test -b 8K -e 64M -f 2 -d fp32 -o sum -p 8For detailed usage instructions of the tool, refer to the Tool Execution section in the Ascend Documentation Center - HCCL Performance Test Tool Guide.View the resultsAfter executing the HCCL Test tool, the display output is as follows:check_result shows success, indicating that the communication operator executed successfully and the AllReduce operator function is correct.aveg_time: The execution time of the collective communication operator, in us.alg_bandwidth: The execution bandwidth of the collective communication operator, in GB/s.data_size: The amount of data participating in collective communication on a single NPU, in Bytes.AppendixOpen Source Third-Party Software DependenciesWhen compiling this project, the following third-party open source software dependencies are required:Open Source SoftwareVersionDownload URLjson3.11.3include.zipmakeself2.5.0makeself-release-2.5.0-patch1.tar.gzopenssl3.0.9openssl-openssl-3.0.9.tar.gzhcomm_utils9.0.0 (aarch64)cann-hcomm-utils_9.0.0_linux-aarch64.tar.gzhcomm_utils9.0.0 (x86_64)cann-hcomm-utils_9.0.0_linux-x86_64.tar.gzgoogletest1.14.0googletest-1.14.0.tar.gzboost1.87.0boost_1_87_0.tar.gzmockcpp2.7-h4mockcpp-2.7.tar.gzmockcpp-patch2.7-h4mockcpp-2.7_py3.patchabseil-cpp20250127.0abseil-cpp-20250127.0.tar.gzprotobuf25.1protobuf-25.1.tar.gzrdma-corev42.7-h1rdma-core-42.7.tar.gzrdma-core-patchv42.7-h1rdma-core-42.7.patchcann-cmakemaster-025cmake-master-025.tar.gz【免费下载链接】hcommHCOMMHuawei Communication是HCCL的通信基础库提供通信域以及通信资源的管理能力。项目地址: https://gitcode.com/cann/hcomm创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考