Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 430447 reports (Page 8289 of 8609)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.X.............X...........] duration: 5.877732866s FAIL (2025-07-09 22:23:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.877758906s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X..X.X...X...X.......X............X.X............X.........X] duration: 1m0.23482575s FAIL (2025-07-09 22:23:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :27.530694236 (397h12m2.301335571s) [upload=317782]: Got next segment. Inline: false Jul 9 22:22:27.530721418 (397h12m2.301362743s) [upload=317782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:22:27.530725656 (397h12m2.301366981s) [upload=317782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:22:27.972697777 (397h12m2.743339102s) [seg_pos={0 0}, upload=317782]: Begin upload segment... Jul 9 22:22:27.972768972 (397h12m2.743410308s) [seg_pos={0 0}, upload=317782]: Done begin upload segment. Jul 9 22:22:27.972793770 (397h12m2.743435105s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973170050 (397h12m2.743811385s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973207692 (397h12m2.743849027s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973462760 (397h12m2.744104095s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973561820 (397h12m2.744203155s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973719421 (397h12m2.744360756s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973862605 (397h12m2.74450394s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.973978998 (397h12m2.744620333s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974101712 (397h12m2.744743047s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974232022 (397h12m2.744873357s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974306995 (397h12m2.74494833s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974500055 (397h12m2.74514139s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974522848 (397h12m2.745164183s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974681611 (397h12m2.745322946s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974766505 (397h12m2.74540783s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.974923444 (397h12m2.745564779s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975010481 (397h12m2.745651816s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:27.975147763 (397h12m2.745789088s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Uploading piece... Jul 9 22:22:29.327909891 (397h12m4.098551226s) [upload=317782]: Next returned nil segment Jul 9 22:22:29.327933787 (397h12m4.098575122s) [upload=317782]: Waiting for error group managing segments... Jul 9 22:22:29.978258832 (397h12m4.748900167s) [noise=true , address=77.90.31.12:4594 , piece_id=LJJXKPHBZKSNCJTOQFDOYBTL4U7ZBSE7RI57EWMI5CZJ5YD5YI3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.979151280 (397h12m4.749792615s) [noise=true , address=77.90.31.21:4894 , piece_id=56U7WC7KQ2S7XLBDNVI7ADMZQX552THVHVK4DANCJF5FJ4XU7TOA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.984195199 (397h12m4.754836534s) [noise=true , address=77.90.31.19:4859 , piece_id=SZQ4CMWGIVVHARJOMKKTN6DI7VAZGVJQPXZKYCTUPPRVU5ZNGCVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.988046737 (397h12m4.758688072s) [noise=true , address=77.90.31.15:4701 , piece_id=T42LFWTYWBBQFFDYHVU6U5YO2H3RUKIJR2DI3U4FSXY4WEU5ZTFA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.992300966 (397h12m4.762942301s) [noise=true , address=77.90.31.16:4749 , piece_id=IUTCAJL2PVYOLFP33YMWX2B5XP2D2FYVR6UBGKNC7QL6OHCMRL6Q, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.996959930 (397h12m4.767601265s) [noise=true , address=77.90.31.17:4781 , piece_id=LCSMEGJZ35OL7TEOKSTMPROLECXWA6MXWDM3QY3GQZ3LBFA4C2UQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:29.997648557 (397h12m4.768289892s) [noise=true , address=77.90.31.14:4690 , piece_id=XT6BVPU5ZEAGNEPCTPXD7D35GBMUHGD2OGGEY2P22QIQM4EAYELQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008095954 (397h12m4.778737289s) [noise=true , address=45.91.116.10:4090 , piece_id=PGHOH3Q7RSNSEOC6QAELKNK6RR5GFURWMOGXEBX7DZ64WXNRD4IA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.008404595 (397h12m4.77904593s) [noise=true , address=77.90.31.13:4649 , piece_id=NYXVP7IV6MQBMUWGECBVUUZ3RYF5XQUXTBMJQFPX3GPA2E4FMTEQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:22:30.392350092 (397h12m5.162991417s) [noise=true , address=79.127.220.97:4371 , piece_id=7C6V6Z6P66SYVSA2NSHUKHPVIV22RS4PASY26INJYPF5DGDDABXA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:24.652397067 (397h12m59.423038402s) [noise=true , address=104.36.82.17:4345 , piece_id=QXC4VIW4NJ5376SZZLSLG62CGH4YUSHO77UQPGFVKBHKALFX6XOQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.282002097 (397h13m1.052643432s) [noise=true , address=104.36.82.35:4593 , piece_id=CP5TQJXAV2OWWXENMFF44P5DI5QU5HIYWAXH525ITTIR4NDTZHLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: <nil> Jul 9 22:23:26.892929169 (397h13m1.663570504s) [noise=true , address=45.91.95.5:4019 , piece_id=WQBPU5OUOFBIT6W6NAUTNFMFI3UU7MBK64U3UET7HN4X7TMCYY3A, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12uYwKKeXmjaZg9GnVTKPvb2h32vj3DHBpJu6vLWruVDZ5TAW8K, address:45.91.95.5:4019): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.892983052 (397h13m1.663624377s) [noise=true , address=104.36.82.19:4429 , piece_id=T6LIMFSHHRXZMQYYXDFTZVZ3GATOMH5LGHCCDLD4MFHK5XU2PSSQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893020182 (397h13m1.663661517s) [noise=true , address=45.91.95.3:4003 , piece_id=OEBEE652D53YZZYWIWIKA7GFVBFIJ2FR5VMNOA7HKNNFWOQBDHVQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:12775mgnBsA3595WztdcNsUBVWCLeeZECij1L2EnwuvzCCtMouk, address:45.91.95.3:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893081459 (397h13m1.663722784s) [noise=true , address=104.36.82.3:4774 , piece_id=4EJYB2DWB736CXZ7CTQOE442AZWQJ2TCDTMMCMKXSANWBUIIABLA, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893095246 (397h13m1.663736581s) [noise=true , address=104.36.82.18:4369 , piece_id=7EW62VOKMH6CXC4CURNXOAS65EXFSIARSR7MXLT2MAS2SJGQWFHQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:23:26.893127879 (397h13m1.663769214s) [noise=true , address=45.91.95.2:4002 , piece_id=N4RIOYFXEIYYJ2DOO3GRRGLPUBNSFDG2OHNXC6TMSYH2KX3TUWXQ, seg_pos={0 0}, upload=317782]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.95.2:4002: use of closed network connection Jul 9 22:23:26.893140864 (397h13m1.663782199s) [seg_pos={0 0}, upload=317782]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 22:23:27.126246135 (397h13m1.89688746s) [upload=317782]: Done uploading segments... Jul 9 22:23:27.126270812 (397h13m1.896912147s) [upload=317782]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.234847683s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X..X....X........X.........X................X...............] duration: 49.942689748s FAIL (2025-07-09 22:23:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.942741519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 4.680785173s FAIL (2025-07-09 22:23:14 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.680819016s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.X.............X..........] duration: 6.040503919s FAIL (2025-07-09 22:22:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.040525531s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.X.............X.........] duration: 6.041491667s FAIL (2025-07-09 22:22:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.041513048s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 4.848221419s FAIL (2025-07-09 22:22:22 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.848250304s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X..X.X...X...X.......X............X.X............X.........] duration: 15.765368046s FAIL (2025-07-09 22:21:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.765395609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 4.687786476s FAIL (2025-07-09 22:21:43 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.687818557s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.X.............X........] duration: 5.936101168s FAIL (2025-07-09 22:21:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.936122268s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X..X....X........X.........X................X..............] duration: 19.47217771s FAIL (2025-07-09 22:21:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.472194162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 4.873741781s FAIL (2025-07-09 22:21:07 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.873770136s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X..X.X...X...X.......X............X.X............X........] duration: 31.456308952s FAIL (2025-07-09 22:20:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.456330864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..X..X....X........X.........X................X.............] duration: 28.417952967s FAIL (2025-07-09 22:20:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.417974538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.X.............X.......] duration: 6.157738334s FAIL (2025-07-09 22:20:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.158091479s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 4.763160195s FAIL (2025-07-09 22:20:41 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.763187196s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...X..X.X...X...X.......X............X.X............X.......] duration: 21.936002035s FAIL (2025-07-09 22:20:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.936043284s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...X..X....X........X.........X................X............] duration: 22.582349093s FAIL (2025-07-09 22:20:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.582430419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.235331835s FAIL (2025-07-09 22:19:38 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.235358957s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.X.............X......] duration: 6.391208496s FAIL (2025-07-09 22:19:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.39123641s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [....X..X.X...X...X.......X............X.X............X......] duration: 34.815314046s FAIL (2025-07-09 22:19:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.815340687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....X..X....X........X.........X................X...........] duration: 26.062519729s FAIL (2025-07-09 22:19:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.062542073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.242946621s FAIL (2025-07-09 22:18:46 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.242975558s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.X.............X.....] duration: 6.081748436s FAIL (2025-07-09 22:18:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.081768004s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....X..X....X........X.........X................X..........] duration: 14.694479768s FAIL (2025-07-09 22:18:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.694493904s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.X.............X....] duration: 6.153151992s FAIL (2025-07-09 22:18:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.153176548s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.215297379s FAIL (2025-07-09 22:18:08 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.215321576s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.....X..X.X...X...X.......X............X.X............X.....] duration: 21.681790962s FAIL (2025-07-09 22:18:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.681808796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.X.............X...] duration: 6.377740971s FAIL (2025-07-09 22:18:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.377764785s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.014273984s FAIL (2025-07-09 22:17:46 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.014292311s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......X..X....X........X.........X................X.........] duration: 20.158953724s FAIL (2025-07-09 22:17:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.158978091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X.X.............X..] duration: 5.906350701s FAIL (2025-07-09 22:17:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.906374677s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [......X..X.X...X...X.......X............X.X............X....] duration: 15.078612639s FAIL (2025-07-09 22:16:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.078629041s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......X..X....X........X.........X................X........] duration: 12.726345961s FAIL (2025-07-09 22:16:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.726358464s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........X..X....X........X.........X................X.......] duration: 45.100331656s FAIL (2025-07-09 22:16:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.10034464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 4.792546389s FAIL (2025-07-09 22:16:37 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.792567729s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.......X..X.X...X...X.......X............X.X............X...] duration: 49.229961155s FAIL (2025-07-09 22:16:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 49.229980442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.X.............X.] duration: 5.968768744s FAIL (2025-07-09 22:16:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.968790836s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 4.710880929s FAIL (2025-07-09 22:16:00 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.710910134s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........X..X....X........X.........X................X......] duration: 43.936730787s FAIL (2025-07-09 22:15:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.936753139s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.X.............X] duration: 30.00047233s FAIL (2025-07-09 22:15:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000492067s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........X..X....X........X.........X................X.....] duration: 15.348104533s FAIL (2025-07-09 22:15:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.348127387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [........X..X.X...X...X.......X............X.X............X..] duration: 24.719128158s FAIL (2025-07-09 22:14:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.719156432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.041658166s FAIL (2025-07-09 22:14:48 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.041701218s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...........X..X....X........X.........X................X....] duration: 31.47768703s FAIL (2025-07-09 22:14:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.477709121s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 6.396113162s FAIL (2025-07-09 22:14:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.39613295s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [............X..X....X........X.........X................X...] duration: 11.430992508s FAIL (2025-07-09 22:13:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.431016203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.........X..X.X...X...X.......X............X.X............X.] duration: 16.780673427s FAIL (2025-07-09 22:13:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17652]: Starting upload Jul 9 22:12:27.306261297 (397h2m2.076902632s) [upload=317652]: Done starting upload Jul 9 22:12:27.306350507 (397h2m2.076991842s) [upload=317652]: Uploading segments... Jul 9 22:12:27.306533027 (397h2m2.077174362s) [upload=317652]: Got next segment. Inline: false Jul 9 22:12:27.306559377 (397h2m2.077200702s) [upload=317652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:12:27.306563665 (397h2m2.077205s) [upload=317652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:12:27.752027654 (397h2m2.522668989s) [seg_pos={0 0}, upload=317652]: Begin upload segment... Jul 9 22:12:27.752088100 (397h2m2.522729425s) [seg_pos={0 0}, upload=317652]: Done begin upload segment. Jul 9 22:12:27.752111415 (397h2m2.52275274s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752139549 (397h2m2.522780884s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752193070 (397h2m2.522834405s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752593206 (397h2m2.523234541s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752612103 (397h2m2.523253438s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.752875005 (397h2m2.52351634s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753103632 (397h2m2.523744967s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753189558 (397h2m2.523830893s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753416151 (397h2m2.524057486s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753634539 (397h2m2.524275874s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.753871572 (397h2m2.524512907s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754080672 (397h2m2.524722007s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754265256 (397h2m2.524906581s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754283209 (397h2m2.524924544s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754451932 (397h2m2.525093267s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754623490 (397h2m2.525264825s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.754709926 (397h2m2.525351261s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:27.755090083 (397h2m2.525731408s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Uploading piece... Jul 9 22:12:28.860648301 (397h2m3.631289636s) [upload=317652]: Next returned nil segment Jul 9 22:12:28.860665795 (397h2m3.63130713s) [upload=317652]: Waiting for error group managing segments... Jul 9 22:12:29.496254650 (397h2m4.266895985s) [noise=true , address=79.127.220.101:4069 , piece_id=SJZSDYUBYPSWPSR25N4JYTTRQVI24ZKZJMZC3BHUZKL7FS4X563A, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.645084533 (397h2m4.415725858s) [noise=true , address=104.36.82.5:4865 , piece_id=E4CF5VDV4TYDCNKKIOVOVHG2RFAHBYNNN2FNBZDKIUFMRY3BSWDQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.743423721 (397h2m4.514065046s) [noise=true , address=79.127.219.47:4048 , piece_id=76JRMPU4HRQGMQ56U5KT5DQKHQ726TXSELFQ6T5NZNSOURJPMGSA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.754935705 (397h2m4.52557704s) [noise=true , address=77.90.31.21:4904 , piece_id=MUS4YKV25TY4IRENIG3BP26A65VP3ZYQGVFDCM4HZQ5G4Z7NYAKQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.757908233 (397h2m4.528549568s) [noise=true , address=77.90.31.11:4582 , piece_id=SYSZXSASM3LZTA2E35JVCNU7MY23CIKKMGPR544C5NO5PVXJTHAA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.771542045 (397h2m4.54218338s) [noise=true , address=45.91.116.9:4286 , piece_id=UPKYD5S543CSRY5LICOTHXOEU7BR3NURZVI7ATSWG4XPJAEFBF7Q, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.775210203 (397h2m4.545851538s) [noise=true , address=77.90.31.13:4655 , piece_id=RUR3J4AM2EYHTUR3WLFRGQ3MJOCD4ZKP57BUNA7KYTJTMBQ4BFRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.854293401 (397h2m4.624934736s) [noise=true , address=79.127.201.215:4334 , piece_id=DV7FXPCR66T5TO6WTTCZ2XU3GKUCC3MJR674FZZIOCRBDOOS4KZQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:29.934847944 (397h2m4.705489279s) [noise=true , address=79.127.219.38:4009 , piece_id=K2E42MFCKDJLAXDNQY33UP74CRMJPMTUEQ6HCBGKHJDFMZYB4LOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:12:30.311500246 (397h2m5.082141581s) [noise=true , address=79.127.205.233:4633 , piece_id=XVTS77RL4T3Z4UNUCTEUAS7KUSRWYYODFVJIN47QXQDWNEAEDZHA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:19.607845263 (397h2m54.378486598s) [noise=true , address=104.36.82.44:4887 , piece_id=5OGXD65OS73PRQ335OVE4YYD4IWPTKGLNPAU7EZUQMXW6P6TROTA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:21.040362882 (397h2m55.811004217s) [noise=true , address=104.36.82.17:4346 , piece_id=NKDDCSX3TBSSYVN3NZRJFRSZQF3L3CYSAPGOQSZM5BHUU4CC44PA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.018239375 (397h3m0.78888071s) [noise=true , address=45.91.95.2:4012 , piece_id=7W6ALZYF3UB5DSKVW6LI4AYSSQTTIS5SHFRLN3WBPTTCAJAWSIVA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: <nil> Jul 9 22:13:26.663521216 (397h3m1.434162551s) [noise=true , address=45.91.95.5:4006 , piece_id=NGTDWG72UQ4PFO5FFCHKOBAEOHPCR5Q4GQGIP6ACXGSBIEXJ2UFQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663581762 (397h3m1.434223097s) [noise=true , address=45.91.95.3:4008 , piece_id=QOWNG3YC6S6YJTF3TV73C5YWLQI7WX2GGHJ4NQMUKBILE6VIMPHQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12HLbheXhir44WzySFWHDdijqzN2XksNaoLLXJfhKDE1aPCrmFZ, address:45.91.95.3:4008): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663617860 (397h3m1.434259195s) [noise=true , address=45.91.95.4:4028 , piece_id=7IZFCOZ7NKU5WMF4HCUOHH25WGFMXCTFTR4I3FAZ4KMNFNUR4CRA, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1D1UakyW46wyk7RwapndBQeDoNa689WfyGZ4Cxk9Qory5zmCbx, address:45.91.95.4:4028): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663747809 (397h3m1.434389144s) [noise=true , address=104.36.82.23:4456 , piece_id=JCFQ6U7PWN5LPMDIGI4IKPIWLB35JGWF4QPDVJK6WGT5DPHECDOQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:104.36.82.23:4456): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:13:26.663832481 (397h3m1.434473806s) [noise=true , address=104.36.82.19:4431 , piece_id=27ZSVGFYCKVLBIK4XFZ63ABGEG3FAUIULPIRGATOHXBZDFVZWXAQ, seg_pos={0 0}, upload=317652]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44474->104.36.82.19:4431: use of closed network connection Jul 9 22:13:26.663846487 (397h3m1.434487822s) [seg_pos={0 0}, upload=317652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:13:26.899088621 (397h3m1.669729956s) [upload=317652]: Done uploading segments... Jul 9 22:13:26.899117085 (397h3m1.66975841s) [upload=317652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.780689818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 4.865555966s FAIL (2025-07-09 22:13:36 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.865585752s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.............X..X....X........X.........X................X..] duration: 50.146319441s FAIL (2025-07-09 22:13:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.146333396s core uplink us-select-1 us1-placement-20

1
...
8287
8288
8289
8290
8291
...
8609