Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X.XX....X....................X..X...XX................] duration: 20.48502213s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:34:38 UTC) orbiter warning: context deadline exceeded Duration: 20.485141918s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001522246s trace: https://jaeger.dev.storj.io/trace/1b2142ff469c06db FAIL (2025-12-16 14:34:38 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001547553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X....X......X......X......................] duration: 8.452836541s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:34:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 8.452854025s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.XX.XX.XXXXX] duration: 5.000945509s trace: https://jaeger.dev.storj.io/trace/62fcad8970210fda FAIL (2025-12-16 14:34:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000973742s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X.XX....X....................X..X...XX...............] duration: 14.300395185s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:33:57 UTC) orbiter warning: context deadline exceeded Duration: 14.300422307s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X....X......X......X.....................] duration: 6.915920903s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:33:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 6.915947224s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X.XX....X....................X..X...XX..............] duration: 17.322512622s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:33:16 UTC) orbiter warning: context deadline exceeded Duration: 17.322544253s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X....X......X......X....................] duration: 7.612900238s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:33:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 7.612931247s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.XX.XX.XXXX] duration: 5.000885873s trace: https://jaeger.dev.storj.io/trace/1a07c6aaf6de29ff FAIL (2025-12-16 14:33:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000915801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X.XX....X....................X..X...XX.............] duration: 24.623205982s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:32:54 UTC) orbiter warning: context deadline exceeded Duration: 24.623240338s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X....X......X......X...................] duration: 8.329074274s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:32:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 8.32910848s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.XX.XX.XXX] duration: 5.001607998s trace: https://jaeger.dev.storj.io/trace/2abb1cfdebb7b864 FAIL (2025-12-16 14:32:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001653054s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........X.XX....X....................X..X...XX............] duration: 20.308492453s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:32:06 UTC) orbiter warning: context deadline exceeded Duration: 20.308668678s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.XX.XX.XX] duration: 5.001839731s trace: https://jaeger.dev.storj.io/trace/65d00a87d3be5ae FAIL (2025-12-16 14:31:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001866622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X....X......X......X..................] duration: 7.975781602s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:31:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 7.975811439s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [X..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.XX.XX.X] duration: 5.001502909s trace: https://jaeger.dev.storj.io/trace/4ac07e382371b520 FAIL (2025-12-16 14:30:55 UTC) orbiter warning: context deadline exceeded Duration: 5.001532745s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........X.XX....X....................X..X...XX...........] duration: 22.160630155s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:30:54 UTC) orbiter warning: context deadline exceeded Duration: 22.160658088s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.XX.XX.] duration: 4.627052648s trace: https://jaeger.dev.storj.io/trace/29dabbc32ceafda8 FAIL (2025-12-16 14:30:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.627078768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X....X......X......X.................] duration: 10.047833538s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:30:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 10.047987231s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X.XX....X....................X..X...XX..........] duration: 22.098743633s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:29:48 UTC) orbiter warning: context deadline exceeded Duration: 22.098767769s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X....X......X......X................] duration: 6.630143738s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:29:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 6.630173896s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.XX.XX] duration: 5.001436271s trace: https://jaeger.dev.storj.io/trace/29dabbc32ceafda8 FAIL (2025-12-16 14:29:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001457572s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............X.XX....X....................X..X...XX.........] duration: 14.34218768s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:29:09 UTC) orbiter warning: context deadline exceeded Duration: 14.342211605s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [X.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.XX.X] duration: 5.002469233s trace: https://jaeger.dev.storj.io/trace/363778be93dff9f0 FAIL (2025-12-16 14:28:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002497006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X.XX....X....................X..X...XX........] duration: 21.064865114s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:28:30 UTC) orbiter warning: context deadline exceeded Duration: 21.065013037s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X....X......X......X...............] duration: 5.716143255s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:28:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 5.716238987s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.XX.] duration: 4.005738511s trace: https://jaeger.dev.storj.io/trace/43ebea7b168a404e FAIL (2025-12-16 14:28:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.005755904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X....X......X......X..............] duration: 6.18823819s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:28:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 6.188268158s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.XX] duration: 5.001307535s trace: https://jaeger.dev.storj.io/trace/43ebea7b168a404e FAIL (2025-12-16 14:27:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001329888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X....X......X......X.............] duration: 5.808672698s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:27:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 5.80870013s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X.XX....X....................X..X...XX.......] duration: 16.810166604s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:27:14 UTC) orbiter warning: context deadline exceeded Duration: 16.810193045s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X.XX....X....................X..X...XX......] duration: 18.528274884s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:26:56 UTC) orbiter warning: context deadline exceeded Duration: 18.528310854s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.X] duration: 5.001341532s trace: https://jaeger.dev.storj.io/trace/3f7966efaf25efaa FAIL (2025-12-16 14:26:45 UTC) orbiter warning: context deadline exceeded Duration: 5.001363155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X....X......X......X............] duration: 5.706030519s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:26:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 5.706060787s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X.XX....X....................X..X...XX.....] duration: 17.546369348s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:26:21 UTC) orbiter warning: context deadline exceeded Duration: 17.546384686s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X.XX....X....................X..X...XX....] duration: 20.483591335s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:26:03 UTC) orbiter warning: context deadline exceeded Duration: 20.483619326s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX.] duration: 3.62909253s trace: https://jaeger.dev.storj.io/trace/33aed6ccaa2a9f55 FAIL (2025-12-16 14:25:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.629121287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X....X......X......X...........] duration: 5.864675754s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:25:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 5.86470487s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXXX] duration: 5.00233182s trace: https://jaeger.dev.storj.io/trace/33aed6ccaa2a9f55 FAIL (2025-12-16 14:25:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002356185s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X.XX....X....................X..X...XX...] duration: 17.421949614s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:25:05 UTC) orbiter warning: context deadline exceeded Duration: 17.421968019s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................X.XX....X....................X..X...XX..] duration: 28.415449346s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:24:47 UTC) orbiter warning: context deadline exceeded Duration: 28.415502857s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [..XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXXX] duration: 5.00091539s trace: https://jaeger.dev.storj.io/trace/54fe49ac00be6702 FAIL (2025-12-16 14:24:47 UTC) orbiter warning: context deadline exceeded Duration: 5.000943073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X....X......X......X..........] duration: 5.853728522s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:24:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 5.85375954s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [...XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXXX] duration: 5.001367876s trace: https://jaeger.dev.storj.io/trace/6fa7668b672660f4 FAIL (2025-12-16 14:24:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001391169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................X.XX....X....................X..X...XX.] duration: 14.332874837s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:24:02 UTC) orbiter warning: context deadline exceeded Duration: 14.332892439s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXXX] duration: 5.000947708s trace: https://jaeger.dev.storj.io/trace/7aa455e2e536c0cd FAIL (2025-12-16 14:23:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000968819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X....X......X......X.........] duration: 5.877346461s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:23:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 5.877373994s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X.XX....X....................X..X...XX] duration: 30.000908248s trace: https://jaeger.dev.storj.io/trace/77285c76fb4fde13 FAIL (2025-12-16 14:23:23 UTC) orbiter warning: context deadline exceeded Duration: 30.000930019s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 33.956716049s trace: https://jaeger.dev.storj.io/trace/ba2c44dde12044 FAIL (2025-12-16 14:23:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=229725]: Uploading piece... Dec 16 13:30:17.600801266 (262h1m17.471884321s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.600906235 (262h1m17.471989281s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601269747 (262h1m17.472352782s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601276790 (262h1m17.472359835s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601630013 (262h1m17.472713048s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601640714 (262h1m17.472723749s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601965091 (262h1m17.473048126s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602047147 (262h1m17.473130192s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602246808 (262h1m17.473329853s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602601232 (262h1m17.473684277s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602716132 (262h1m17.473799167s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602996997 (262h1m17.474080042s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603027615 (262h1m17.47411065s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603255378 (262h1m17.474338413s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603501387 (262h1m17.474584432s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603959158 (262h1m17.475042203s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.604485590 (262h1m17.475568635s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:19.111089229 (262h1m18.982172264s) [upload=229725]: Next returned nil segment Dec 16 13:30:19.111104708 (262h1m18.982187753s) [upload=229725]: Waiting for error group managing segments... Dec 16 13:30:19.527483020 (262h1m19.398566075s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.621753699 (262h1m19.492836744s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.670564204 (262h1m19.541647249s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.672302895 (262h1m19.54338594s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.716239789 (262h1m19.587322834s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.735358003 (262h1m19.606441048s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.750731367 (262h1m19.621814412s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.764604575 (262h1m19.63568761s) [noise=true , address=45.91.116.7:4029 , piece_id=NZ5HRBRJ3QMNZMXJLKD5NPC3ZZFAS7VIDQNXPURRPWOCMPTXFUNQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.796960486 (262h1m19.668043531s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.812007538 (262h1m19.683090573s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.032271213 (262h1m19.903354248s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.099742841 (262h1m19.970825886s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.231931353 (262h1m20.103014398s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394678143 (262h1m20.265761188s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394706487 (262h1m20.265789532s) [seg_pos={0 0}, upload=229725]: Segment reached optimal threshold of 14 pieces. Dec 16 13:30:20.394971411 (262h1m20.266054466s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.394981671 (262h1m20.266064716s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395367956 (262h1m20.266451001s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395378747 (262h1m20.266461792s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395542348 (262h1m20.266625393s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395565903 (262h1m20.266648948s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395688968 (262h1m20.266772013s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395695741 (262h1m20.266778786s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395718144 (262h1m20.266801189s) [seg_pos={0 0}, upload=229725]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 13:30:20.395862569 (262h1m20.266945604s) [upload=229725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:30:20.395896854 (262h1m20.266979899s) [upload=229725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:30:20.679088774 (262h1m20.550171819s) [upload=229725]: Done uploading segments... Dec 16 13:30:20.679109594 (262h1m20.550192629s) [upload=229725]: Upload finished. err: <nil> Duration: 33.956734413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [X....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XXXXXX] duration: 5.000698166s trace: https://jaeger.dev.storj.io/trace/1ac083fdbb6a30f5 FAIL (2025-12-16 14:23:05 UTC) orbiter warning: context deadline exceeded Duration: 5.000751659s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)