Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 332064 reports (Page 6320 of 6642)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X............X.X............X.........X..................] duration: 21.605179895s FAIL (2025-07-09 22:37:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 21.605196235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X.........X................X.....................X.........] duration: 16.872995031s FAIL (2025-07-09 22:37:13 UTC) orbiter warning: context deadline exceeded Duration: 16.873014629s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.X.............X.........................X] duration: 30.000828897s FAIL (2025-07-09 22:37:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000848575s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....X............X.X............X.........X.................] duration: 18.178149734s FAIL (2025-07-09 22:36:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 18.178175663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 4.941683957s FAIL (2025-07-09 22:36:15 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.941704475s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X.........X................X.....................X........] duration: 11.670153354s FAIL (2025-07-09 22:36:07 UTC) orbiter warning: context deadline exceeded Duration: 11.670174685s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.X.............X.........................] duration: 10.810362154s FAIL (2025-07-09 22:35:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.810382433s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....X............X.X............X.........X................] duration: 22.363025639s FAIL (2025-07-09 22:35:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 22.363042931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 4.780695603s FAIL (2025-07-09 22:35:28 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.780722757s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X.........X................X.....................X.......] duration: 46.99339731s FAIL (2025-07-09 22:35:27 UTC) orbiter warning: context deadline exceeded Duration: 46.993421837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......X............X.X............X.........X...............] duration: 41.278936781s FAIL (2025-07-09 22:35:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 41.2789631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.X.............X........................] duration: 6.187153545s FAIL (2025-07-09 22:34:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.187177942s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 4.795764129s FAIL (2025-07-09 22:34:50 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.795793034s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......X............X.X............X.........X..............] duration: 47.88380963s FAIL (2025-07-09 22:34:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 47.883840187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X.........X................X.....................X......] duration: 16.601780199s FAIL (2025-07-09 22:34:07 UTC) orbiter warning: context deadline exceeded Duration: 16.601796099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.X.............X.......................] duration: 5.892041813s FAIL (2025-07-09 22:34:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.89209752s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 4.850934882s FAIL (2025-07-09 22:33:58 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.850958887s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X.........X................X.....................X.....] duration: 26.682262349s FAIL (2025-07-09 22:33:51 UTC) orbiter warning: context deadline exceeded Duration: 26.682306113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 4.756119526s FAIL (2025-07-09 22:32:57 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.756146148s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.X.............X......................] duration: 5.985268162s FAIL (2025-07-09 22:32:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.985289043s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X.........X................X.....................X....] duration: 17.634138457s FAIL (2025-07-09 22:32:53 UTC) orbiter warning: context deadline exceeded Duration: 17.634157222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X.......X............X.X............X.........X.............] duration: 12.329731702s FAIL (2025-07-09 22:32:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 12.3297619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 4.849740337s FAIL (2025-07-09 22:32:14 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.849775605s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X.......X............X.X............X.........X............] duration: 46.214805545s FAIL (2025-07-09 22:32:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 46.214853337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X.........X................X.....................X...] duration: 24.548154859s FAIL (2025-07-09 22:31:49 UTC) orbiter warning: context deadline exceeded Duration: 24.548180177s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.X.............X.....................] duration: 6.248787443s FAIL (2025-07-09 22:31:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.248810637s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 4.883523687s FAIL (2025-07-09 22:31:21 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.883543503s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X.........X................X.....................X..] duration: 32.904892922s FAIL (2025-07-09 22:31:17 UTC) orbiter warning: context deadline exceeded Duration: 32.904914032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 4.914741189s FAIL (2025-07-09 22:31:04 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.914768802s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.X.............X....................] duration: 14.283714789s FAIL (2025-07-09 22:31:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.283738787s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..X.......X............X.X............X.........X...........] duration: 28.278909509s FAIL (2025-07-09 22:30:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 28.278955106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...X.......X............X.X............X.........X..........] duration: 26.348675406s FAIL (2025-07-09 22:30:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 26.34869924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X........X.........X................X.....................X.] duration: 13.682375079s FAIL (2025-07-09 22:30:06 UTC) orbiter warning: context deadline exceeded Duration: 13.682388554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 4.91029728s FAIL (2025-07-09 22:29:58 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.910325203s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X........X.........X................X.....................X] duration: 1m0.000492849s FAIL (2025-07-09 22:29:52 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000513979s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.X.............X...................] duration: 6.506165499s FAIL (2025-07-09 22:29:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.506352327s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.947563406s FAIL (2025-07-09 22:29:31 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.947587482s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X...X.......X............X.X............X.........X.........] duration: 13.850932317s FAIL (2025-07-09 22:29:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 13.850948117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.X...X.......X............X.X............X.........X........] duration: 25.555385628s FAIL (2025-07-09 22:28:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 25.555402661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 4.956425827s FAIL (2025-07-09 22:28:48 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.956449522s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.X.............X..................] duration: 6.223628812s FAIL (2025-07-09 22:28:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.223647417s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..X...X.......X............X.X............X.........X.......] duration: 47.14138246s FAIL (2025-07-09 22:28:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 47.14139796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 4.771190531s FAIL (2025-07-09 22:28:31 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.77121722s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.X.............X.................] duration: 6.217659522s FAIL (2025-07-09 22:28:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.217675002s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X........X.........X................X.....................] duration: 21.459600755s FAIL (2025-07-09 22:28:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.459621834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...X...X.......X............X.X............X.........X......] duration: 50.154506264s FAIL (2025-07-09 22:27:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 50.154528277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.094087964s FAIL (2025-07-09 22:27:31 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.094111458s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X........X.........X................X....................] duration: 29.249455966s FAIL (2025-07-09 22:27:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.249480744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.X.............X................] duration: 6.325583319s FAIL (2025-07-09 22:27:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.325606255s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 4.834884005s FAIL (2025-07-09 22:26:52 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.834912228s core uplink eu1

1
...
6318
6319
6320
6321
6322
...
6642