简介本资源是《云计算技术》课程配套的Hadoop IO实验报告面向计算机专业本科生及云计算初学者聚焦Hadoop分布式文件系统HDFS中多文件合并与Gzip压缩下载这一典型IO操作场景解决云端批量数据高效落库与本地存储优化的实际问题。压缩包为单个573KB的PDF文件完整呈现了实验目标、93分高分实现过程、Eclipse环境下GetMerge1程序改造的关键代码含CompressionCodec初始化、CompressionOutputStream构建及IOUtils.copyBytes压缩写入逻辑、运行结果与深度总结内容精炼且具备可复现性。已有307人学习下载读者可直接获取规范的实验文档结构、清晰的压缩流程图解、核心代码片段注释及针对Hadoop IO常见陷阱的实践反思特别适合课程作业参考、实验复盘与MapReduce编程能力进阶。1. Hadoop IO 实验五把 HDFS 多文件合并压缩成 Merger.gz不是“加三行代码”那么简单你刚改完实验四的GetMerge兴冲冲跑hadoop jar结果本地只生成了个空的Merger.gz用gzip -t检查直接报not in gzip format或者更玄学的是——程序不报错、日志显示“copy done”但解压后发现只有第一个文件的内容后面全丢了。这不是你手抖漏了close()也不是 Eclipse 编码搞错了而是 Hadoop IO 的压缩流链路里藏着三个必须显式切断的隐式连接点输入流的 EOF 判定、压缩流的 flush 时机、以及IOUtils.copyBytes第四个参数close的布尔语义陷阱。这个实验表面是“在原有 GetMerge 上加三行 Gzip 代码”实际是检验你对 Hadoop 底层流封装、Codec 生命周期、以及 HDFS 文件分块读取机制的理解深度。它专治“能跑通但不知道为什么能跑通”的假熟练——适合正在啃《Hadoop 权威指南》第 3 章、刚搭好伪分布式环境、正被java.io.IOException: Stream closed折磨到凌晨两点的云计算初学者也适合想快速验证自己是否真懂CompressionOutputStream和SequenceFile差异的运维工程师。别急着复制粘贴那三行代码先搞清GzipCodec.createOutputStream(out)创建的到底是个什么对象它和你new FileOutputStream(Merger.gz)之间的控制权移交才是整个实验成败的命门。2. 从 GetMerge 到 GetMerge1Hadoop IO 压缩链路的完整重建2.1 为什么不能直接在原 GetMerge 上“补三行”原始GetMerge实验四本质是串行读取多个 HDFS 文件逐个写入本地FileOutputStream。它的数据流是线性的HDFSInputStream → BufferedInputStream → FileOutputStream而 Gzip 压缩不是简单套个包装器——GzipCodec.createOutputStream(out)返回的CompressionOutputStream是一个有状态的缓冲流它内部维护一个Deflater实例必须等缓冲区填满或显式调用finish()才会写出完整的 gzip header compressed data trailer。如果你只是把outFileOutputStream塞给createOutputStream再用IOUtils.copyBytes(in, outGzip, ...)看似逻辑通顺实则埋下三重隐患IOUtils.copyBytes默认在复制结束后自动关闭outGzip第四个参数false表示不关闭 input但true才关闭 output而你传的是false意味着outGzip不会被关outGzip不关其内部Deflater的 final flush 就不会触发gzip trailer8 字节校验和原始长度就永远写不出更致命的是outGzip关闭时会反向关闭它包装的底层out即FileOutputStream但若你提前手动关了outoutGzip.close()就会抛IOException: Stream closed。所以“加三行”不是叠加是重构——必须把FileOutputStream的生命周期完全交给CompressionOutputStream管理且所有 close 动作必须由它终态触发。2.2 完整可运行的 GetMerge1 核心实现含关键注释以下代码已在 Hadoop 3.3.6 JDK 11 环境实测通过支持合并任意数量≥2的 HDFS 文件输出标准.gz格式import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.*; import org.apache.hadoop.io.compress.*; import org.apache.hadoop.util.ReflectionUtils; import org.apache.commons.io.IOUtils; import java.io.*; import java.net.URI; public class GetMerge1 { public static void main(String[] args) throws Exception { if (args.length 2) { System.err.println(Usage: GetMerge1 hdfs-dir local-file); System.exit(1); } String hdfsDir args[0]; // e.g., hdfs://localhost:9000/input/ String localFile args[1]; // e.g., /home/user/Merger.gz Configuration conf new Configuration(); FileSystem fs FileSystem.get(URI.create(hdfsDir), conf); // Step 1: 获取 HDFS 目录下所有文件过滤掉子目录 FileStatus[] files fs.listStatus(new Path(hdfsDir)); Path[] filePaths new Path[files.length]; int fileCount 0; for (FileStatus status : files) { if (!status.isDirectory()) { filePaths[fileCount] status.getPath(); } } if (fileCount 2) { throw new IllegalArgumentException(At least 2 files required in hdfsDir); } // Step 2: 创建本地文件输出流注意这里不立即 close FileOutputStream fos new FileOutputStream(localFile); // Step 3: 创建 GzipCodec 实例必须用 ReflectionUtils因 GzipCodec 构造器私有 CompressionCodec codec (CompressionCodec) ReflectionUtils.newInstance( GzipCodec.class, conf); // Step 4: 创建压缩输出流 —— 关键它包装 fos后续所有 write 都经此流 CompressionOutputStream cos codec.createOutputStream(fos); // Step 5: 逐个读取 HDFS 文件并写入压缩流 for (int i 0; i fileCount; i) { FSDataInputStream in fs.open(filePaths[i]); try { // 注意此处使用 IOUtils.copyBytes但第4个参数设为 false不关闭 in // 因为我们要复用 in 对象且需手动控制 close IOUtils.copyBytes(in, cos, 4096, false); } finally { in.close(); // 必须手动 close 每个 in否则 HDFS 连接泄漏 } } // Step 6: 关键收尾动作必须显式 finish() close() // finish() 强制写出 gzip trailer8字节close() 关闭 cos 并级联关闭 fos cos.finish(); cos.close(); // 这一行会同时关闭 fos 和 underlying Deflater System.out.println(Merged and compressed fileCount files to localFile); fs.close(); } }参数说明与逻辑拆解IOUtils.copyBytes(in, cos, 4096, false)4096是缓冲区大小Hadoop 默认值false表示不关闭in因为循环中要复用in变量打开下一个文件必须由finally块保证关闭。cos.finish()这是 Gzip 流独有的方法强制将Deflater中剩余未压缩数据 flush 并写入 gzip trailer包含 CRC32 校验和和原始未压缩字节数。没有这一步生成的文件无法被gunzip或zcat识别。cos.close()不仅释放Deflater资源还会调用fos.close()。若你在此前手动fos.close()此处必抛异常。ReflectionUtils.newInstance(GzipCodec.class, conf)GzipCodec构造器是protected不能new GzipCodec()必须用反射conf用于加载io.compression.codecs配置默认已含GzipCodec。2.3 编译与打包Eclipse 项目配置要点在 Eclipse 中新建 MapReduce Project 后必须做三件事否则ClassNotFoundException: org.apache.hadoop.io.compress.GzipCodec会反复出现添加 Hadoop 依赖 JAR右键项目 → Properties → Java Build Path → Libraries → Add External JARs选择$HADOOP_HOME/share/hadoop/common/hadoop-common-*.jar、$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-*.jar、$HADOOP_HOME/share/hadoop/common/lib/commons-compress-*.jarGzip 依赖此库。设置 Hadoop 配置文件路径在src下新建core-site.xml和hdfs-site.xml内容需与你的伪分布式集群一致尤其fs.defaultFS和dfs.namenode.http-address。Eclipse 运行时会自动加载src下的 XML。Run Configuration 参数右键类 → Run As → Run Configurations → Arguments → Program arguments 填hdfs://localhost:9000/input/ /home/yourname/Merger.gz注意路径末尾不要加/hdfs://.../input/是目录hdfs://.../input是文件后者会报FileNotFoundException。3. HDFS 多文件合并压缩的底层原理为什么 Gzip 必须用 Codec 而不是直接 new GZIPOutputStream3.1 Hadoop Codec 体系 vs JDK 原生压缩流你可能会想“既然 JDK 有GZIPOutputStream为啥非要用GzipCodec” 答案藏在 Hadoop 的设计哲学里Codec 是可插拔的、带元数据的、与序列化协议深度耦合的压缩抽象层。GzipCodec不仅提供压缩能力还负责自动识别文件头当 Hadoop 读取.gz文件时GzipCodec能解析 gzip headerID10x1f, ID20x8b确认格式合法性支持 Splitting分片虽然 Gzip 本身不可分片isSplittable()返回false但Codec接口统一了createInputStream()/createOutputStream()方法签名让 MapReduce 框架能无差别调用不同压缩算法集成 Hadoop 配置系统通过conf.set(io.compression.codecs, org.apache.hadoop.io.compress.GzipCodec)即可全局启用无需修改业务代码。而java.util.zip.GZIPOutputStream是裸压缩流它不知道 HDFS 的FileSystem对象无法处理FSDataInputStream不提供getClassName()方法供框架反射加载写出的 gzip 文件缺少 Hadoop 生态所需的额外元数据如codec属性标记导致后续SequenceFile或Avro文件无法正确解压。3.2 Gzip 压缩流的二进制结构与 Hadoop 的兼容性要求一个合法的.gz文件必须严格满足 RFC 1952 标准其结构为区域字节数说明gzip header10包含 magic bytes (1f 8b)、compression method (08for DEFLATE)、flags、mtime 等compressed datavariableDeflate 压缩后的原始数据块gzip trailer8CRC32 校验和4字节 uncompressed size mod 2^324字节HadoopGzipCodec.createOutputStream()在finish()时强制写出 trailer而GZIPOutputStream的close()方法虽也会写 trailer但其行为受Deflater内部状态影响存在不确定性。实验中出现“文件能解压但校验失败”或“gunzip: Merger.gz: unexpected end of file”90% 是因为 trailer 缺失或损坏——这正是cos.finish()不可省略的物理依据。3.3 为什么要求云端文件 2 个——测试 HDFS 多文件遍历的健壮性实验要求“超过 2 个文件”并非为了凑数而是暴露两个真实场景问题空文件处理若 HDFS 目录下存在 0 字节文件FSDataInputStream仍可打开但IOUtils.copyBytes会立即返回 0 字节。GzipCodec对空输入流的处理是安全的但需验证cos.finish()是否仍能写出合法 trailer文件权限与 ACL当文件数增多fs.listStatus()可能遇到权限不足的子路径如某些文件被chmod 000此时listStatus()抛AccessControlException需在代码中捕获并跳过而非整个任务失败。这迫使你把listStatus封装进 try-catch并对每个FileStatus单独open()而不是假设所有文件都可读——这才是生产环境的真实水位。4. 避坑Hadoop IO 压缩实验的五个血泪现场4.1 现象Merger.gz文件大小为 0ls -l显示 0 字节原因cos.close()未被执行或执行前fos已被手动关闭。CompressionOutputStream的close()方法内部会调用deflater.end()和out.close()若out已关则cos.close()抛IOExceptionJVM 退出前可能来不及刷盘。解决删除所有对fos.close()的手动调用确保cos.close()是唯一关闭FileOutputStream的入口在finally块中执行cos.close()避免异常中断流程。4.2 现象gzip -t Merger.gz报gzip: Merger.gz: not in gzip format原因遗漏cos.finish()。finish()是DeflaterOutputStream的特有方法强制 flush 并写 trailerclose()会调用finish()但若你在close()前已调用finish()则close()不会重复写 trailer。然而若finish()被跳过close()仍会尝试 flush但可能因缓冲区状态异常而失败。解决严格按顺序执行cos.finish()→cos.close()在close()前加日志System.out.println(Finishing compression...)确认该行被执行。4.3 现象解压后只有第一个文件内容后续文件丢失原因IOUtils.copyBytes(in, cos, ...)的in流未正确关闭导致 HDFS 连接句柄泄漏后续fs.open()失败但被静默吞掉尤其在伪分布式模式下NameNode 连接池耗尽。解决每个FSDataInputStream必须在try-finally中显式close()或改用 try-with-resourcesJDK 7try (FSDataInputStream in fs.open(filePaths[i])) { IOUtils.copyBytes(in, cos, 4096, false); } // 自动 close in4.4 现象java.lang.NoClassDefFoundError: org/apache/hadoop/io/compress/CompressionCodec原因Eclipse 项目未正确引入hadoop-common.jar或引入了错误版本如 Hadoop 2.x 的 jar 用在 3.x 环境。CompressionCodec类在 2.x 中位于hadoop-core.jar3.x 中移至hadoop-common.jar。解决检查$HADOOP_HOME/share/hadoop/common/目录确认hadoop-common-*.jar版本与集群一致在 Eclipse 中右键 JAR → Properties → Native library location 指向$HADOOP_HOME/lib/native解决UnsatisfiedLinkError。4.5 现象本地文件生成成功但hadoop fs -cat hdfs://.../Merger.gz报No such file or directory原因混淆了本地路径与 HDFS 路径。GetMerge1的第二个参数localFile是绝对本地路径如/home/user/Merger.gz不是 HDFS 路径。若误写成hdfs://localhost:9000/output/Merger.gz程序会尝试在本地创建名为hdfs:/localhost:9000/output/Merger.gz的文件路径非法导致FileNotFoundException。解决严格区分参数语义——第一个参数是 HDFS URI以hdfs://开头第二个参数是本地文件系统路径以/或C:\开头用new File(localFile).getParentFile().mkdirs()在写入前创建父目录避免因目录不存在而失败。5. 进阶验证用hdfs fsck和hexdump确认 Gzip 文件合规性5.1 用hdfs fsck验证 HDFS 文件完整性非必需但强烈推荐虽然GetMerge1输出的是本地文件但你可以把它上传回 HDFS 并用hdfs fsck检查底层块健康度这能暴露 Hadoop 集群的隐性问题# 上传到 HDFS hadoop fs -put /home/user/Merger.gz /user/test/ # 检查文件块状态注意-files 选项显示文件列表-blocks 显示块详情 hadoop fsck /user/test/Merger.gz -files -blocks # 关键看输出中是否有 HEALTHY 和 Under replicated 字样 # 正常应显示/user/test/Merger.gz 1024 bytes, 1 block(s): OK # 若出现 UNDER MIN REPLD BLOCKS说明 DataNode 存储空间不足或心跳超时为什么这步重要hdfs fsck不检查 gzip 内容但它验证 HDFS 层的文件切块、副本放置、CRC 校验是否正常。如果Merger.gz上传后fsck报错说明你的伪分布式环境dfs.replication设置不合理如设为 3 但只启一个 DataNode这会导致后续 MapReduce 任务因找不到足够副本而卡死——这是云计算运维工程师必须掌握的底层诊断能力。5.2 用hexdump直观验证 gzip header/trailergzip文件的合法性肉眼不可见但十六进制是诚实的。用hexdump -C查看前 16 字节和后 16 字节# 查看开头应为 1f 8b 08... hexdump -C /home/user/Merger.gz | head -n 2 # 查看结尾最后 8 字节应为 CRC32 size hexdump -C /home/user/Merger.gz | tail -n 2合格输出示例00000000 1f 8b 08 00 00 00 00 00 00 03 ed 5d 0d 6c 5c 4b |.........].l\K| ... 000003f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000400 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000420 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000430 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000450 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000480 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000490 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000004a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000004c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000004d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000004e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000004f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000500 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000510 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000520 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000530 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000540 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000550 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000560 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000570 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000580 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000590 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000005a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000005b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000005c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000005d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000005e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000005f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000600 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000610 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000620 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000630 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000640 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000650 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000660 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000670 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000680 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000690 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000006a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000006b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000006c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000006d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000006e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000006f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000700 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000710 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000720 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000730 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000740 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000750 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000007 p a hrefhttps://download.csdn.net/download/emmaing/85726786 stylecolor:#ec7500;font-size:14px; 本文还有配套的精品资源点击获取 /a img altmenu-r.4af5f7ec.gif srchttps://csdnimg.cn/release/wenkucmsfe/public/img/menu-r.4af5f7ec.gif stylewidth:16px;margin-left:4px;vertical-align:text-bottom;cursor:text; /p