Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 433073 reports (Page 8341 of 8662)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..X...X.......X............X.X............X.........X.......] duration: 47.14138246s FAIL (2025-07-09 22:28:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 47.14139796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 4.771190531s FAIL (2025-07-09 22:28:31 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.77121722s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.X.............X.................] duration: 6.217659522s FAIL (2025-07-09 22:28:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.217675002s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X........X.........X................X.....................] duration: 21.459600755s FAIL (2025-07-09 22:28:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.459621834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...X...X.......X............X.X............X.........X......] duration: 50.154506264s FAIL (2025-07-09 22:27:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 50.154528277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.094087964s FAIL (2025-07-09 22:27:31 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.094111458s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X........X.........X................X....................] duration: 29.249455966s FAIL (2025-07-09 22:27:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.249480744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.X.............X................] duration: 6.325583319s FAIL (2025-07-09 22:27:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.325606255s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 4.834884005s FAIL (2025-07-09 22:26:52 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.834912228s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.X.............X...............] duration: 6.22287017s FAIL (2025-07-09 22:26:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.22289077s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X...X...X.......X............X.X............X.........X.....] duration: 29.373642057s FAIL (2025-07-09 22:26:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 29.373665161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 4.945249146s FAIL (2025-07-09 22:26:31 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.945283239s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X........X.........X................X...................] duration: 28.454815982s FAIL (2025-07-09 22:26:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.454827735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 4.774706178s FAIL (2025-07-09 22:26:00 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.774732289s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X....X........X.........X................X..................] duration: 21.756006145s FAIL (2025-07-09 22:25:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.756027206s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.X.............X..............] duration: 6.226725401s FAIL (2025-07-09 22:25:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.226744648s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X...X...X.......X............X.X............X.........X....] duration: 13.463134126s FAIL (2025-07-09 22:25:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 13.463159704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X....X........X.........X................X.................] duration: 24.960768128s FAIL (2025-07-09 22:25:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.960791392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.032007068s FAIL (2025-07-09 22:24:58 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.032031355s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X.X...X...X.......X............X.X............X.........X...] duration: 12.337737412s FAIL (2025-07-09 22:24:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 12.337764755s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.X.............X.............] duration: 16.226788578s FAIL (2025-07-09 22:24:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.226811522s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X.X...X...X.......X............X.X............X.........X..] duration: 20.964432904s FAIL (2025-07-09 22:24:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 20.964448164s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..X....X........X.........X................X................] duration: 41.399036305s FAIL (2025-07-09 22:24:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.399055801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.X.............X............] duration: 6.06491421s FAIL (2025-07-09 22:24:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.064939519s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X.X...X...X.......X............X.X............X.........X.] duration: 35.429644354s FAIL (2025-07-09 22:24:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 35.42967852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 4.79210472s FAIL (2025-07-09 22:23:52 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.792127884s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.X.............X...........] duration: 5.877732866s FAIL (2025-07-09 22:23:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.877758906s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X..X.X...X...X.......X............X.X............X.........X] duration: 1m0.23482575s FAIL (2025-07-09 22:23:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.234847683s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X..X....X........X.........X................X...............] duration: 49.942689748s FAIL (2025-07-09 22:23:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.942741519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 4.680785173s FAIL (2025-07-09 22:23:14 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.680819016s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.X.............X..........] duration: 6.040503919s FAIL (2025-07-09 22:22:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.040525531s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.X.............X.........] duration: 6.041491667s FAIL (2025-07-09 22:22:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.041513048s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 4.848221419s FAIL (2025-07-09 22:22:22 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.848250304s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X..X.X...X...X.......X............X.X............X.........] duration: 15.765368046s FAIL (2025-07-09 22:21:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.765395609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 4.687786476s FAIL (2025-07-09 22:21:43 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.687818557s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.X.............X........] duration: 5.936101168s FAIL (2025-07-09 22:21:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.936122268s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X..X....X........X.........X................X..............] duration: 19.47217771s FAIL (2025-07-09 22:21:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.472194162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 4.873741781s FAIL (2025-07-09 22:21:07 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.873770136s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X..X.X...X...X.......X............X.X............X........] duration: 31.456308952s FAIL (2025-07-09 22:20:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.456330864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..X..X....X........X.........X................X.............] duration: 28.417952967s FAIL (2025-07-09 22:20:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.417974538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.X.............X.......] duration: 6.157738334s FAIL (2025-07-09 22:20:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.158091479s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 4.763160195s FAIL (2025-07-09 22:20:41 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.763187196s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...X..X.X...X...X.......X............X.X............X.......] duration: 21.936002035s FAIL (2025-07-09 22:20:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.936043284s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...X..X....X........X.........X................X............] duration: 22.582349093s FAIL (2025-07-09 22:20:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.582430419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.235331835s FAIL (2025-07-09 22:19:38 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.235358957s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.X.............X......] duration: 6.391208496s FAIL (2025-07-09 22:19:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.39123641s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [....X..X.X...X...X.......X............X.X............X......] duration: 34.815314046s FAIL (2025-07-09 22:19:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.815340687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....X..X....X........X.........X................X...........] duration: 26.062519729s FAIL (2025-07-09 22:19:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.062542073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.242946621s FAIL (2025-07-09 22:18:46 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.242975558s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.X.............X.....] duration: 6.081748436s FAIL (2025-07-09 22:18:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.081768004s edge gateway us-select-1

1
...
8339
8340
8341
8342
8343
...
8662