Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 389805 reports (Page 7474 of 7797)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X................X.....................X...................] duration: 22.500220915s FAIL (2025-07-09 22:45:09 UTC) orbiter warning: context deadline exceeded Duration: 22.500280108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X.X............X.........X............................] duration: 22.260854126s FAIL (2025-07-09 22:45:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 22.260889974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 4.786578163s FAIL (2025-07-09 22:44:47 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.786603712s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X.X............X.........X...........................] duration: 14.466863618s FAIL (2025-07-09 22:44:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 14.466887073s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X................X.....................X..................] duration: 27.902613537s FAIL (2025-07-09 22:44:10 UTC) orbiter warning: context deadline exceeded Duration: 27.902636441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 4.991849069s FAIL (2025-07-09 22:44:09 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.9919911s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.X.............X.........................X...........] duration: 5.938627232s FAIL (2025-07-09 22:44:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.938652641s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X.X............X.........X..........................] duration: 37.130709623s FAIL (2025-07-09 22:43:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 37.130739018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.X.............X.........................X..........] duration: 6.081547869s FAIL (2025-07-09 22:43:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.081571475s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 4.858961992s FAIL (2025-07-09 22:42:57 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.859042144s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X.X............X.........X.........................] duration: 22.038519604s FAIL (2025-07-09 22:42:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 22.038546546s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X................X.....................X.................] duration: 18.23819047s FAIL (2025-07-09 22:42:49 UTC) orbiter warning: context deadline exceeded Duration: 18.238213605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.X.............X.........................X.........] duration: 6.181723275s FAIL (2025-07-09 22:42:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.181744714s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.X.............X.........................X........] duration: 8.025023862s FAIL (2025-07-09 22:42:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.025045212s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X................X.....................X................] duration: 15.099246257s FAIL (2025-07-09 22:42:04 UTC) orbiter warning: context deadline exceeded Duration: 15.099265905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X.X............X.........X........................] duration: 15.401489121s FAIL (2025-07-09 22:42:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 15.401514099s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 4.906007182s FAIL (2025-07-09 22:41:57 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.906035436s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X.X............X.........X.......................] duration: 14.480889841s FAIL (2025-07-09 22:41:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 14.48091556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 4.98268003s FAIL (2025-07-09 22:41:24 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.982707242s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.X.............X.........................X.......] duration: 6.202391768s FAIL (2025-07-09 22:41:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.202412787s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X.X............X.........X......................] duration: 36.25208788s FAIL (2025-07-09 22:41:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 36.252136343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 4.800786884s FAIL (2025-07-09 22:41:02 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.800816s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X................X.....................X...............] duration: 22.967101501s FAIL (2025-07-09 22:40:58 UTC) orbiter warning: context deadline exceeded Duration: 22.967113715s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X................X.....................X..............] duration: 35.346954528s FAIL (2025-07-09 22:40:35 UTC) orbiter warning: context deadline exceeded Duration: 35.346977181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 4.875063966s FAIL (2025-07-09 22:40:30 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.875103512s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.X.............X.........................X......] duration: 6.746479412s FAIL (2025-07-09 22:40:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.746501072s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.X.............X.........................X.....] duration: 6.222342335s FAIL (2025-07-09 22:39:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.222365919s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X............X.X............X.........X.....................] duration: 15.650027035s FAIL (2025-07-09 22:39:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 15.650051952s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X................X.....................X.............] duration: 11.324584015s FAIL (2025-07-09 22:39:30 UTC) orbiter warning: context deadline exceeded Duration: 11.324595937s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.X.............X.........................X....] duration: 5.862729601s FAIL (2025-07-09 22:39:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.862754439s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 4.765219282s FAIL (2025-07-09 22:39:19 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.765245173s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X................X.....................X............] duration: 36.118048362s FAIL (2025-07-09 22:39:18 UTC) orbiter warning: context deadline exceeded Duration: 36.118058521s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.X.............X.........................X...] duration: 7.27442175s FAIL (2025-07-09 22:39:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.274443161s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X................X.....................X...........] duration: 21.192600847s FAIL (2025-07-09 22:38:42 UTC) orbiter warning: context deadline exceeded Duration: 21.192621988s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.X.............X.........................X..] duration: 8.877939061s FAIL (2025-07-09 22:38:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.877968067s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X............X.X............X.........X....................] duration: 17.49462725s FAIL (2025-07-09 22:38:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 17.494650576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 4.803473444s FAIL (2025-07-09 22:38:29 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.803496126s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.X.............X.........................X.] duration: 6.453819398s FAIL (2025-07-09 22:37:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.453843884s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..X............X.X............X.........X...................] duration: 14.900615171s FAIL (2025-07-09 22:37:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 14.900652201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X.........X................X.....................X..........] duration: 13.725432978s FAIL (2025-07-09 22:37:42 UTC) orbiter warning: context deadline exceeded Duration: 13.725450201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.196212703s FAIL (2025-07-09 22:37:29 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.196240193s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X............X.X............X.........X..................] duration: 21.605179895s FAIL (2025-07-09 22:37:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 21.605196235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X.........X................X.....................X.........] duration: 16.872995031s FAIL (2025-07-09 22:37:13 UTC) orbiter warning: context deadline exceeded Duration: 16.873014629s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.X.............X.........................X] duration: 30.000828897s FAIL (2025-07-09 22:37:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000848575s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....X............X.X............X.........X.................] duration: 18.178149734s FAIL (2025-07-09 22:36:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 18.178175663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 4.941683957s FAIL (2025-07-09 22:36:15 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.941704475s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X.........X................X.....................X........] duration: 11.670153354s FAIL (2025-07-09 22:36:07 UTC) orbiter warning: context deadline exceeded Duration: 11.670174685s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.X.............X.........................] duration: 10.810362154s FAIL (2025-07-09 22:35:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.810382433s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....X............X.X............X.........X................] duration: 22.363025639s FAIL (2025-07-09 22:35:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 22.363042931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 4.780695603s FAIL (2025-07-09 22:35:28 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.780722757s core uplink eu1

1
...
7472
7473
7474
7475
7476
...
7797