Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...........................................X..] duration: 18.23662742s trace: https://jaeger.dev.storj.io/trace/4fea8f5179350142 FAIL (2025-12-10 13:22:49 UTC) orbiter warning: context deadline exceeded Duration: 18.236694477s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.064475741s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:22:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 5.06450063s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXX] duration: 5.001154553s trace: https://jaeger.dev.storj.io/trace/58539274a4b4a9c6 FAIL (2025-12-10 13:22:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001178308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X...X.................................X..........] duration: 11.16191454s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:22:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 11.16200387s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [X.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXX] duration: 5.001124428s trace: https://jaeger.dev.storj.io/trace/6a287dfeb67b675f FAIL (2025-12-10 13:22:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001151228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 3.828734234s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:22:09 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 3.828767908s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X...X.................................X.........] duration: 11.868779098s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:22:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 11.868805858s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...........................................X.] duration: 29.95316374s trace: https://jaeger.dev.storj.io/trace/4fea8f5179350142 FAIL (2025-12-10 13:22:02 UTC) orbiter warning: context deadline exceeded Duration: 29.953196622s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 12.953369388s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:21:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 12.953393123s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 7.458079406s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:21:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 7.458112929s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X...X.................................X........] duration: 19.597423619s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:21:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 19.597454799s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XX] duration: 5.002516502s trace: https://jaeger.dev.storj.io/trace/1aa2fd1f17178546 FAIL (2025-12-10 13:21:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002541108s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...........................................X] duration: 30.000947823s trace: https://jaeger.dev.storj.io/trace/4fea8f5179350142 FAIL (2025-12-10 13:20:48 UTC) orbiter warning: context deadline exceeded Duration: 30.000980385s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X...X.................................X.......] duration: 18.88991633s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:20:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 18.889945265s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 7.611791362s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:20:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 7.611821719s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [.XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.X] duration: 5.004073864s trace: https://jaeger.dev.storj.io/trace/16983f505205df88 FAIL (2025-12-10 13:20:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.004116335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X...X.................................X......] duration: 8.810821542s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:19:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 8.810848613s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.89272718s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:19:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 6.892814538s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 25.767534168s trace: https://jaeger.dev.storj.io/trace/42b65ff8650ecdb9 FAIL (2025-12-10 13:19:39 UTC) orbiter warning: context deadline exceeded Duration: 25.767565889s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [X.XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.] duration: 5.001335338s trace: https://jaeger.dev.storj.io/trace/30df20ac45c75b5f FAIL (2025-12-10 13:19:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001349845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X...X.................................X.....] duration: 9.671969114s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:19:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 9.671996516s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 9.169276242s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:19:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 9.169310609s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XX.XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX] duration: 5.001825762s trace: https://jaeger.dev.storj.io/trace/30df20ac45c75b5f FAIL (2025-12-10 13:19:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001853094s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 15.151121695s trace: https://jaeger.dev.storj.io/trace/42b65ff8650ecdb9 FAIL (2025-12-10 13:18:55 UTC) orbiter warning: context deadline exceeded Duration: 15.151170226s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X...X.................................X....] duration: 21.66561991s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:18:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 21.665645378s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXX.XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXX] duration: 5.001357633s trace: https://jaeger.dev.storj.io/trace/2b97876e6a118823 FAIL (2025-12-10 13:18:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001373765s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.079001118s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:18:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 7.079030274s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 22.687486784s trace: https://jaeger.dev.storj.io/trace/42b65ff8650ecdb9 FAIL (2025-12-10 13:18:16 UTC) orbiter warning: context deadline exceeded Duration: 22.687524224s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [.XXX.XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXX] duration: 5.001458284s trace: https://jaeger.dev.storj.io/trace/5eb3ccf5053cbb11 FAIL (2025-12-10 13:17:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001486199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.131224936s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:17:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 5.131252369s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 20.685762663s trace: https://jaeger.dev.storj.io/trace/42b65ff8650ecdb9 FAIL (2025-12-10 13:17:28 UTC) orbiter warning: context deadline exceeded Duration: 20.685788321s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X...X.................................X...] duration: 11.88228576s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:17:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 11.882307671s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [X.XXX.XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXX] duration: 5.001225859s trace: https://jaeger.dev.storj.io/trace/475e99f97b74e9cd FAIL (2025-12-10 13:17:19 UTC) orbiter warning: context deadline exceeded Duration: 5.001260104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X...X.................................X..] duration: 14.464915545s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:17:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 14.464943529s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X...X.................................X.] duration: 13.481862424s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:16:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 13.48189215s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XX.XXX.XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXX] duration: 5.001214469s trace: https://jaeger.dev.storj.io/trace/589200b17a2f1a23 FAIL (2025-12-10 13:16:28 UTC) orbiter warning: context deadline exceeded Duration: 5.001242032s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 11.342766799s trace: https://jaeger.dev.storj.io/trace/42b65ff8650ecdb9 FAIL (2025-12-10 13:16:26 UTC) orbiter warning: context deadline exceeded Duration: 11.342794052s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 11.759692215s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:16:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 11.759719506s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXX.XXX.XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXX] duration: 5.001980739s trace: https://jaeger.dev.storj.io/trace/1e1c9db16cfe4595 FAIL (2025-12-10 13:16:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002002521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X...X.................................X] duration: 30.002120802s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:16:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 30.002148145s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 28.962777251s trace: https://jaeger.dev.storj.io/trace/42b65ff8650ecdb9 FAIL (2025-12-10 13:15:41 UTC) orbiter warning: context deadline exceeded Duration: 28.962797118s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.25960394s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:15:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 6.259628577s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 26.27517392s trace: https://jaeger.dev.storj.io/trace/42b65ff8650ecdb9 FAIL (2025-12-10 13:15:06 UTC) orbiter warning: context deadline exceeded Duration: 26.275208767s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXXX.XXX.XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXX] duration: 5.001677248s trace: https://jaeger.dev.storj.io/trace/25c085d1d36831a6 FAIL (2025-12-10 13:15:06 UTC) orbiter warning: context deadline exceeded Duration: 5.001701003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 11.834586576s trace: https://jaeger.dev.storj.io/trace/3e617b7e8b619dc8 FAIL (2025-12-10 13:14:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 10 12:51:04.674984490 (117h22m4.546067535s) [upload=102438]: Starting upload Dec 10 12:51:04.675072718 (117h22m4.546155763s) [upload=102438]: Done starting upload Dec 10 12:51:04.675114728 (117h22m4.546197783s) [upload=102438]: Uploading segments... Dec 10 12:51:04.675369533 (117h22m4.546452578s) [upload=102438]: Got next segment. Inline: false Dec 10 12:51:04.675410801 (117h22m4.546493846s) [upload=102438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 12:51:04.675418727 (117h22m4.546501772s) [upload=102438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 12:51:04.919523043 (117h22m4.790606078s) [seg_pos={0 0}, upload=102438]: Begin upload segment... Dec 10 12:51:04.919578539 (117h22m4.790661574s) [seg_pos={0 0}, upload=102438]: Done begin upload segment. Dec 10 12:51:04.919602203 (117h22m4.790685238s) [noise=true , address=145.79.227.6:4259 , piece_id=VPGTQ7PM7JXWA4ST2WM7XNCGGRFVK4JTZQDZLSRZK3SZSUOLSEDA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.919616100 (117h22m4.790699135s) [noise=true , address=145.79.227.31:4797 , piece_id=EBA5MTUCJL5FBGMWIV3KLW4KD4VBL53CSWPCJDBPO667CTBCWYMA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.919656296 (117h22m4.790739342s) [noise=true , address=145.79.227.30:4762 , piece_id=AFIKUYYRD4TAQVKDAMQ5BONILSVODKFIDJY2BPT3LR3H423VHRPQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.920013957 (117h22m4.791096992s) [noise=true , address=145.79.227.26:4653 , piece_id=E46SYQBUSM2LGLGKVX4E5KNYKH2UC7TOSMYPWNFB22OFP6GIKKUQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.920433486 (117h22m4.791516531s) [noise=true , address=45.91.116.4:4937 , piece_id=55YSQLKNIYGEDGAQ6TTPMNKCWPPCEBWQEIHK3AFF7CYKKPYQQ6DQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.920940111 (117h22m4.792023156s) [noise=true , address=77.90.31.10:4518 , piece_id=4DKESX63HP5M5OVFC5S76YITXX5ZLT2YBQ3DUDFAYPO4I2HXHMHQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.921295698 (117h22m4.792378743s) [noise=true , address=145.79.227.39:4013 , piece_id=IWZAPKSEZP4QKXAEGBEZTCKCMQZ2FKK3HC63QOM42CGRMKMDKQCQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.921897213 (117h22m4.792980258s) [noise=true , address=77.90.31.17:4772 , piece_id=6U5P5QCRCKLFKBKDP7WZEAOPARFYCZENU4FXYXFJQUE2QOE7SLLA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.922433946 (117h22m4.793517001s) [noise=true , address=145.79.227.28:4727 , piece_id=ZI3RGFE2FGZHBVWWOHGBYORDAKNAR3RLTNMY2CSSINX6SQHUXUVA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.922924670 (117h22m4.794007715s) [noise=true , address=145.79.227.18:4006 , piece_id=3BPSY5GXYJB2AIJMJ5NSEPL3L2YNRTPTAPPH7F6HR6ZWXRPLUWYQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.923387381 (117h22m4.794470456s) [noise=true , address=45.91.116.5:4297 , piece_id=CAHFTXFOIDLYX5E63NF6QM4VC4EZVX44YFRX64YUTHSASS5FEURQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.924021308 (117h22m4.795104343s) [noise=true , address=77.90.31.11:4555 , piece_id=7AMGITWEZS5POWW5NXHJWSW356DYOOSXZLER6TYX6ZIWWUB5U2ZA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.924118784 (117h22m4.795201829s) [noise=true , address=45.91.116.11:4119 , piece_id=SEHCN63GGGZLSOOCFZ7XVRXTEBKCBKJEQCE3WUPDF5HEWMYVZ5QA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.924508225 (117h22m4.79559127s) [noise=true , address=145.79.227.39:4015 , piece_id=QI4BWPMWT5J4EV2GNDK6UQPOF7HJXL5Y533NSC3XURAGVI3LEPMQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.924854635 (117h22m4.79593768s) [noise=true , address=145.79.227.34:4905 , piece_id=5PBEPDIGJ6FF6CYH6EZVRATVDATS6VDZVTM7KFPEDERNZTFNLHQQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.924961718 (117h22m4.796044764s) [noise=true , address=145.79.227.32:4835 , piece_id=2MSV5IURJGCWCKNSXNNSQCLCKRKCXGIWSQD4L37LVUMVMOJ6IMEQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:06.795802854 (117h22m6.666885899s) [upload=102438]: Next returned nil segment Dec 10 12:51:06.795819325 (117h22m6.66690237s) [upload=102438]: Waiting for error group managing segments... Dec 10 12:51:07.171503576 (117h22m7.042586621s) [noise=true , address=45.91.116.4:4937 , piece_id=55YSQLKNIYGEDGAQ6TTPMNKCWPPCEBWQEIHK3AFF7CYKKPYQQ6DQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:07.302120748 (117h22m7.173203783s) [noise=true , address=77.90.31.10:4518 , piece_id=4DKESX63HP5M5OVFC5S76YITXX5ZLT2YBQ3DUDFAYPO4I2HXHMHQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:07.953880125 (117h22m7.82496317s) [noise=true , address=45.91.116.11:4119 , piece_id=SEHCN63GGGZLSOOCFZ7XVRXTEBKCBKJEQCE3WUPDF5HEWMYVZ5QA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:08.330402092 (117h22m8.201485127s) [noise=true , address=77.90.31.17:4772 , piece_id=6U5P5QCRCKLFKBKDP7WZEAOPARFYCZENU4FXYXFJQUE2QOE7SLLA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:08.354924307 (117h22m8.226007342s) [noise=true , address=145.79.227.18:4006 , piece_id=3BPSY5GXYJB2AIJMJ5NSEPL3L2YNRTPTAPPH7F6HR6ZWXRPLUWYQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:08.367868335 (117h22m8.23895138s) [noise=true , address=145.79.227.28:4727 , piece_id=ZI3RGFE2FGZHBVWWOHGBYORDAKNAR3RLTNMY2CSSINX6SQHUXUVA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:08.459684813 (117h22m8.330767848s) [noise=true , address=45.91.116.5:4297 , piece_id=CAHFTXFOIDLYX5E63NF6QM4VC4EZVX44YFRX64YUTHSASS5FEURQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:09.359805502 (117h22m9.230888547s) [noise=true , address=145.79.227.32:4835 , piece_id=2MSV5IURJGCWCKNSXNNSQCLCKRKCXGIWSQD4L37LVUMVMOJ6IMEQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:09.403023688 (117h22m9.274106733s) [noise=true , address=77.90.31.11:4555 , piece_id=7AMGITWEZS5POWW5NXHJWSW356DYOOSXZLER6TYX6ZIWWUB5U2ZA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:10.377693002 (117h22m10.248776038s) [noise=true , address=145.79.227.31:4797 , piece_id=EBA5MTUCJL5FBGMWIV3KLW4KD4VBL53CSWPCJDBPO667CTBCWYMA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:10.640943968 (117h22m10.512027013s) [noise=true , address=145.79.227.39:4015 , piece_id=QI4BWPMWT5J4EV2GNDK6UQPOF7HJXL5Y533NSC3XURAGVI3LEPMQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:10.715995092 (117h22m10.587078127s) [noise=true , address=145.79.227.26:4653 , piece_id=E46SYQBUSM2LGLGKVX4E5KNYKH2UC7TOSMYPWNFB22OFP6GIKKUQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:11.196423518 (117h22m11.067506553s) [noise=true , address=145.79.227.34:4905 , piece_id=5PBEPDIGJ6FF6CYH6EZVRATVDATS6VDZVTM7KFPEDERNZTFNLHQQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:11.196444558 (117h22m11.067527593s) [seg_pos={0 0}, upload=102438]: Segment reached optimal threshold of 13 pieces. Dec 10 12:51:11.196665599 (117h22m11.067748634s) [noise=true , address=145.79.227.6:4259 , piece_id=VPGTQ7PM7JXWA4ST2WM7XNCGGRFVK4JTZQDZLSRZK3SZSUOLSEDA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Dec 10 12:51:11.196673764 (117h22m11.0677568s) [noise=true , address=145.79.227.6:4259 , piece_id=VPGTQ7PM7JXWA4ST2WM7XNCGGRFVK4JTZQDZLSRZK3SZSUOLSEDA, seg_pos={0 0}, upload=102438]: Long tail cancellation due to optimal threshold Dec 10 12:51:11.196948347 (117h22m11.068031382s) [noise=true , address=145.79.227.39:4013 , piece_id=IWZAPKSEZP4QKXAEGBEZTCKCMQZ2FKK3HC63QOM42CGRMKMDKQCQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: context canceled; context canceled Dec 10 12:51:11.196956242 (117h22m11.068039277s) [noise=true , address=145.79.227.39:4013 , piece_id=IWZAPKSEZP4QKXAEGBEZTCKCMQZ2FKK3HC63QOM42CGRMKMDKQCQ, seg_pos={0 0}, upload=102438]: Long tail cancellation due to optimal threshold Dec 10 12:51:11.197050211 (117h22m11.068133257s) [noise=true , address=145.79.227.30:4762 , piece_id=AFIKUYYRD4TAQVKDAMQ5BONILSVODKFIDJY2BPT3LR3H423VHRPQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 10 12:51:11.197081912 (117h22m11.068164957s) [noise=true , address=145.79.227.30:4762 , piece_id=AFIKUYYRD4TAQVKDAMQ5BONILSVODKFIDJY2BPT3LR3H423VHRPQ, seg_pos={0 0}, upload=102438]: Long tail cancellation due to optimal threshold Dec 10 12:51:11.197112329 (117h22m11.068195364s) [seg_pos={0 0}, upload=102438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 12:51:11.197212730 (117h22m11.068295775s) [upload=102438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 12:51:11.197220476 (117h22m11.068303511s) [upload=102438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 12:51:11.512593572 (117h22m11.383676617s) [upload=102438]: Done uploading segments... Dec 10 12:51:11.512614722 (117h22m11.383697757s) [upload=102438]: Upload finished. err: <nil> Duration: 11.834611023s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXXXX.XXX.XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXX] duration: 5.001220892s trace: https://jaeger.dev.storj.io/trace/2a189cf12a614762 FAIL (2025-12-10 13:14:33 UTC) orbiter warning: context deadline exceeded Duration: 5.001250417s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 10.122146173s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:14:27 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 10.122172464s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 9.35627103s trace: https://jaeger.dev.storj.io/trace/42b65ff8650ecdb9 FAIL (2025-12-10 13:14:11 UTC) orbiter warning: context deadline exceeded Duration: 9.356284637s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 20.350662749s trace: https://jaeger.dev.storj.io/trace/3e617b7e8b619dc8 FAIL (2025-12-10 13:13:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 10 12:51:04.674984490 (117h22m4.546067535s) [upload=102438]: Starting upload Dec 10 12:51:04.675072718 (117h22m4.546155763s) [upload=102438]: Done starting upload Dec 10 12:51:04.675114728 (117h22m4.546197783s) [upload=102438]: Uploading segments... Dec 10 12:51:04.675369533 (117h22m4.546452578s) [upload=102438]: Got next segment. Inline: false Dec 10 12:51:04.675410801 (117h22m4.546493846s) [upload=102438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 12:51:04.675418727 (117h22m4.546501772s) [upload=102438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 12:51:04.919523043 (117h22m4.790606078s) [seg_pos={0 0}, upload=102438]: Begin upload segment... Dec 10 12:51:04.919578539 (117h22m4.790661574s) [seg_pos={0 0}, upload=102438]: Done begin upload segment. Dec 10 12:51:04.919602203 (117h22m4.790685238s) [noise=true , address=145.79.227.6:4259 , piece_id=VPGTQ7PM7JXWA4ST2WM7XNCGGRFVK4JTZQDZLSRZK3SZSUOLSEDA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.919616100 (117h22m4.790699135s) [noise=true , address=145.79.227.31:4797 , piece_id=EBA5MTUCJL5FBGMWIV3KLW4KD4VBL53CSWPCJDBPO667CTBCWYMA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.919656296 (117h22m4.790739342s) [noise=true , address=145.79.227.30:4762 , piece_id=AFIKUYYRD4TAQVKDAMQ5BONILSVODKFIDJY2BPT3LR3H423VHRPQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.920013957 (117h22m4.791096992s) [noise=true , address=145.79.227.26:4653 , piece_id=E46SYQBUSM2LGLGKVX4E5KNYKH2UC7TOSMYPWNFB22OFP6GIKKUQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.920433486 (117h22m4.791516531s) [noise=true , address=45.91.116.4:4937 , piece_id=55YSQLKNIYGEDGAQ6TTPMNKCWPPCEBWQEIHK3AFF7CYKKPYQQ6DQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.920940111 (117h22m4.792023156s) [noise=true , address=77.90.31.10:4518 , piece_id=4DKESX63HP5M5OVFC5S76YITXX5ZLT2YBQ3DUDFAYPO4I2HXHMHQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.921295698 (117h22m4.792378743s) [noise=true , address=145.79.227.39:4013 , piece_id=IWZAPKSEZP4QKXAEGBEZTCKCMQZ2FKK3HC63QOM42CGRMKMDKQCQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.921897213 (117h22m4.792980258s) [noise=true , address=77.90.31.17:4772 , piece_id=6U5P5QCRCKLFKBKDP7WZEAOPARFYCZENU4FXYXFJQUE2QOE7SLLA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.922433946 (117h22m4.793517001s) [noise=true , address=145.79.227.28:4727 , piece_id=ZI3RGFE2FGZHBVWWOHGBYORDAKNAR3RLTNMY2CSSINX6SQHUXUVA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.922924670 (117h22m4.794007715s) [noise=true , address=145.79.227.18:4006 , piece_id=3BPSY5GXYJB2AIJMJ5NSEPL3L2YNRTPTAPPH7F6HR6ZWXRPLUWYQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.923387381 (117h22m4.794470456s) [noise=true , address=45.91.116.5:4297 , piece_id=CAHFTXFOIDLYX5E63NF6QM4VC4EZVX44YFRX64YUTHSASS5FEURQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.924021308 (117h22m4.795104343s) [noise=true , address=77.90.31.11:4555 , piece_id=7AMGITWEZS5POWW5NXHJWSW356DYOOSXZLER6TYX6ZIWWUB5U2ZA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.924118784 (117h22m4.795201829s) [noise=true , address=45.91.116.11:4119 , piece_id=SEHCN63GGGZLSOOCFZ7XVRXTEBKCBKJEQCE3WUPDF5HEWMYVZ5QA, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.924508225 (117h22m4.79559127s) [noise=true , address=145.79.227.39:4015 , piece_id=QI4BWPMWT5J4EV2GNDK6UQPOF7HJXL5Y533NSC3XURAGVI3LEPMQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.924854635 (117h22m4.79593768s) [noise=true , address=145.79.227.34:4905 , piece_id=5PBEPDIGJ6FF6CYH6EZVRATVDATS6VDZVTM7KFPEDERNZTFNLHQQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:04.924961718 (117h22m4.796044764s) [noise=true , address=145.79.227.32:4835 , piece_id=2MSV5IURJGCWCKNSXNNSQCLCKRKCXGIWSQD4L37LVUMVMOJ6IMEQ, seg_pos={0 0}, upload=102438]: Uploading piece... Dec 10 12:51:06.795802854 (117h22m6.666885899s) [upload=102438]: Next returned nil segment Dec 10 12:51:06.795819325 (117h22m6.66690237s) [upload=102438]: Waiting for error group managing segments... Dec 10 12:51:07.171503576 (117h22m7.042586621s) [noise=true , address=45.91.116.4:4937 , piece_id=55YSQLKNIYGEDGAQ6TTPMNKCWPPCEBWQEIHK3AFF7CYKKPYQQ6DQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:07.302120748 (117h22m7.173203783s) [noise=true , address=77.90.31.10:4518 , piece_id=4DKESX63HP5M5OVFC5S76YITXX5ZLT2YBQ3DUDFAYPO4I2HXHMHQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:07.953880125 (117h22m7.82496317s) [noise=true , address=45.91.116.11:4119 , piece_id=SEHCN63GGGZLSOOCFZ7XVRXTEBKCBKJEQCE3WUPDF5HEWMYVZ5QA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:08.330402092 (117h22m8.201485127s) [noise=true , address=77.90.31.17:4772 , piece_id=6U5P5QCRCKLFKBKDP7WZEAOPARFYCZENU4FXYXFJQUE2QOE7SLLA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:08.354924307 (117h22m8.226007342s) [noise=true , address=145.79.227.18:4006 , piece_id=3BPSY5GXYJB2AIJMJ5NSEPL3L2YNRTPTAPPH7F6HR6ZWXRPLUWYQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:08.367868335 (117h22m8.23895138s) [noise=true , address=145.79.227.28:4727 , piece_id=ZI3RGFE2FGZHBVWWOHGBYORDAKNAR3RLTNMY2CSSINX6SQHUXUVA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:08.459684813 (117h22m8.330767848s) [noise=true , address=45.91.116.5:4297 , piece_id=CAHFTXFOIDLYX5E63NF6QM4VC4EZVX44YFRX64YUTHSASS5FEURQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:09.359805502 (117h22m9.230888547s) [noise=true , address=145.79.227.32:4835 , piece_id=2MSV5IURJGCWCKNSXNNSQCLCKRKCXGIWSQD4L37LVUMVMOJ6IMEQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:09.403023688 (117h22m9.274106733s) [noise=true , address=77.90.31.11:4555 , piece_id=7AMGITWEZS5POWW5NXHJWSW356DYOOSXZLER6TYX6ZIWWUB5U2ZA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:10.377693002 (117h22m10.248776038s) [noise=true , address=145.79.227.31:4797 , piece_id=EBA5MTUCJL5FBGMWIV3KLW4KD4VBL53CSWPCJDBPO667CTBCWYMA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:10.640943968 (117h22m10.512027013s) [noise=true , address=145.79.227.39:4015 , piece_id=QI4BWPMWT5J4EV2GNDK6UQPOF7HJXL5Y533NSC3XURAGVI3LEPMQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:10.715995092 (117h22m10.587078127s) [noise=true , address=145.79.227.26:4653 , piece_id=E46SYQBUSM2LGLGKVX4E5KNYKH2UC7TOSMYPWNFB22OFP6GIKKUQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:11.196423518 (117h22m11.067506553s) [noise=true , address=145.79.227.34:4905 , piece_id=5PBEPDIGJ6FF6CYH6EZVRATVDATS6VDZVTM7KFPEDERNZTFNLHQQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: <nil> Dec 10 12:51:11.196444558 (117h22m11.067527593s) [seg_pos={0 0}, upload=102438]: Segment reached optimal threshold of 13 pieces. Dec 10 12:51:11.196665599 (117h22m11.067748634s) [noise=true , address=145.79.227.6:4259 , piece_id=VPGTQ7PM7JXWA4ST2WM7XNCGGRFVK4JTZQDZLSRZK3SZSUOLSEDA, seg_pos={0 0}, upload=102438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Dec 10 12:51:11.196673764 (117h22m11.0677568s) [noise=true , address=145.79.227.6:4259 , piece_id=VPGTQ7PM7JXWA4ST2WM7XNCGGRFVK4JTZQDZLSRZK3SZSUOLSEDA, seg_pos={0 0}, upload=102438]: Long tail cancellation due to optimal threshold Dec 10 12:51:11.196948347 (117h22m11.068031382s) [noise=true , address=145.79.227.39:4013 , piece_id=IWZAPKSEZP4QKXAEGBEZTCKCMQZ2FKK3HC63QOM42CGRMKMDKQCQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: context canceled; context canceled Dec 10 12:51:11.196956242 (117h22m11.068039277s) [noise=true , address=145.79.227.39:4013 , piece_id=IWZAPKSEZP4QKXAEGBEZTCKCMQZ2FKK3HC63QOM42CGRMKMDKQCQ, seg_pos={0 0}, upload=102438]: Long tail cancellation due to optimal threshold Dec 10 12:51:11.197050211 (117h22m11.068133257s) [noise=true , address=145.79.227.30:4762 , piece_id=AFIKUYYRD4TAQVKDAMQ5BONILSVODKFIDJY2BPT3LR3H423VHRPQ, seg_pos={0 0}, upload=102438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 10 12:51:11.197081912 (117h22m11.068164957s) [noise=true , address=145.79.227.30:4762 , piece_id=AFIKUYYRD4TAQVKDAMQ5BONILSVODKFIDJY2BPT3LR3H423VHRPQ, seg_pos={0 0}, upload=102438]: Long tail cancellation due to optimal threshold Dec 10 12:51:11.197112329 (117h22m11.068195364s) [seg_pos={0 0}, upload=102438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 12:51:11.197212730 (117h22m11.068295775s) [upload=102438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 12:51:11.197220476 (117h22m11.068303511s) [upload=102438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 12:51:11.512593572 (117h22m11.383676617s) [upload=102438]: Done uploading segments... Dec 10 12:51:11.512614722 (117h22m11.383697757s) [upload=102438]: Upload finished. err: <nil> Duration: 20.350695422s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXXXXX.XXX.XX.XXXXX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXX] duration: 5.001134551s trace: https://jaeger.dev.storj.io/trace/2d32c7bf96478db2 FAIL (2025-12-10 13:13:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00115013s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)